Extended reality support for shoppers

ABSTRACT

Systems and methods are provided for retail environments. In one implementation, a non-transitory computer-readable medium may include instructions that when executed by a processor cause the processor to perform a method. The method may include obtaining an electronic shopping list associated with a customer, the electronic shopping list identifying a target product; automatically determining that the target product is unavailable at a particular retail store; automatically identify an alternative product, available at the particular retail store, as a potential alternative to the target product; receiving at least one image representative of at least a portion of the particular retail store wherein the at least one image includes a representation of the alternative product; and analyzing the at least one image to cause an extended reality appliance to present to a shopper a visual indicator associated with the alternative product in the particular retail store.

CROSS REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of priority of U.S. ProvisionalApplication No. 63/183,872, filed on May 4, 2021; U.S. ProvisionalApplication No. 63/230,215, filed on Aug. 6, 2021; and U.S. ProvisionalApplication No. 63/254,290, filed on Oct. 11, 2021.

The foregoing applications are incorporated herein by reference in theirentirety.

BACKGROUND Technical Field

The present disclosure relates generally to systems, methods, anddevices for identifying products in retail stores, and more specificallyto systems, methods, and devices for capturing, collecting, andautomatically analyzing images of products displayed in retail storesfor purposes of providing one or more functions associated with theidentified products.

Background Information

Shopping in stores is a prevalent part of modern daily life. Storeowners (also known as “retailers”) stock a wide variety of products onstore shelves and add associated labels and promotions to the storeshelves. Typically, retailers have a set of processes and instructionsfor organizing products on the store shelves. The source of some ofthese instructions may include contractual obligations and otherpreferences related to the retailer methodology for placement ofproducts on the store shelves. Nowadays, many retailers and supplierssend people to stores to personally monitor compliance with the desiredproduct placement. Such a monitoring technique, however, may beinefficient and may result in nonuniform compliance among retailersrelative to various product-related guidelines. This technique may alsoresult in significant gaps in compliance, as it does not allow forcontinuous monitoring of dynamically changing product displays. Toincrease productivity, among other potential benefits, there is atechnological need to provide a dynamic solution that will automaticallymonitor retail spaces. Such a solution, for example and among otherfeatures, may automatically determine whether a disparity exists betweena desired product placement and an actual product placement.

The disclosed devices and methods are directed to providing new ways formonitoring retail establishments using image processing and supportingsensors.

SUMMARY

Embodiments consistent with the present disclosure provide systems,methods, and devices for capturing, collecting, and analyzing images ofproducts displayed in retail stores. For example, consistent with thedisclosed embodiments, an example system may receive an image depictinga store shelf having products displayed thereon, identify the productson the store shelf, and trigger an alert when disparity exists betweenthe desired product placement and the actual product placement.

In an embodiment, a non-transitory computer-readable medium may includeinstructions that when executed by a processor cause the processor toperform a method. The method may include obtaining an electronicshopping list associated with a customer, the electronic shopping listidentifying a target product; automatically determining that the targetproduct is unavailable at a particular retail store; automaticallyidentify an alternative product, available at the particular retailstore, as a potential alternative to the target product; receiving atleast one image representative of at least a portion of the particularretail store wherein the at least one image includes a representation ofthe alternative product; and analyzing the at least one image to causean extended reality appliance to present to a shopper a visual indicatorassociated with the alternative product in the particular retail store.

In an embodiment, a system for providing augmented reality support toretail store pickers may include at least one computer processorconfigured to execute one or more operating instructions of a computerprogram located in a non-transitory computer readable medium to obtainan electronic shopping list associated with a customer, the electronicshopping list identifying a target product; automatically determine thatthe target product is unavailable at a particular retail store;automatically identify an alternative product, available at theparticular retail store, as a potential alternative to the targetproduct; receive at least one image representative of at least a portionof the particular retail store wherein the at least one image includes arepresentation of the alternative product; and analyze the at least oneimage to cause an extended reality appliance to present to a shopper avisual indicator associated with the alternative product in theparticular retail store.

In an embodiment, a method of locating an available alternative productin a store may include the steps of using a computer processor toexecute one or more operating instructions of a computer program locatedin a non-transitory computer readable medium to obtain an electronicshopping list associated with a customer, the electronic shopping listidentifying a target product; automatically determine that the targetproduct is unavailable at a particular retail store; automaticallyidentify an alternative product, available at the particular retailstore, as a potential alternative to the target product; receive atleast one image representative of at least a portion of the particularretail store wherein the at least one image includes a representation ofthe alternative product; and analyze the at least one image to cause anextended reality appliance to present to a shopper a visual indicatorassociated with the alternative product in the particular retail store.

In an embodiment, a non-transitory computer-readable medium may includeinstructions that when executed by a processor cause the processor toperform a method for facilitating online shopping for customers inphysical retail stores. The method may include making a firstdetermination that a customer located in a physical retail store hasengaged in at least one action associated with an interest in a categoryof products associated with the retail store; making a seconddetermination that the at least one action is associated with a targetproduct, included in the category of products, that is out of stock inthe physical retail store; and in response to the first and seconddeterminations, providing a notification to the customer, wherein thenotification includes information regarding ordering the target productvia an online platform.

In an embodiment, a system for facilitating customer access to retailproducts may include at least one processing unit configured to performthe steps of making a first determination that a customer located in aphysical retail store has engaged in at least one action associated withan interest in a category of products associated with the retail store;making a second determination that the at least one action is associatedwith a target product, included in the category of products, that is outof stock in the physical retail store; and in response to the first andsecond determinations, providing a notification to the customer, whereinthe notification includes information regarding ordering the targetproduct via an online platform.

In an embodiment, a method for facilitating customer access to retailproducts may include making a first determination that a customerlocated in a physical retail store has engaged in at least one actionassociated with an interest in a category of products associated withthe retail store; making a second determination that the at least oneaction is associated with a target product, included in the category ofproducts, that is out of stock in the physical retail store; and inresponse to the first and second determinations, providing anotification to the customer, wherein the notification includesinformation regarding ordering the target product via an onlineplatform.

In an embodiment, a non-transitory computer-readable medium may includeinstructions that when executed by at least one processor cause the atleast one processor to perform a method for distributed shopping. Themethod includes accessing a virtual shopping list associated with auser, the virtual shopping list identifying a plurality of products. Afirst indicator associated with a prospective visit by the user to afirst retail store is determined. A second indicator associated with aprospective visit by the user to a second retail store is determined.Pricing information for the plurality of products located at the firstretail store and at the second retail store is received. Inventoryinformation for the plurality of products located at the first retailstore and at the second retail store is received. An updated virtualshopping list is automatically generated, including an indication of alocation for purchasing each of the plurality of products. Theindication of the location is associated with the first retail store orthe second retail store. The updated virtual shopping list is generatedbased on a combination of the received pricing information and thereceived inventory information. The updated virtual shopping list isprovided to the user.

In an embodiment, a system for distributing shopping may include atleast one processing unit configured to perform the steps of: accessinga virtual shopping list associated with a user, the virtual shoppinglist identifying a plurality of products; determining a first indicatorassociated with a prospective visit by the user to a first retail store;determining a second indicator associated with a prospective visit bythe user to a second retail store; receiving pricing information for theplurality of products located at the first retail store and at thesecond retail store; receiving inventory information for the pluralityof products located at the first retail store and at the second retailstore; automatically generating an updated virtual shopping listincluding an indication of a location for purchasing each of theplurality of products, the indication of the location being associatedwith the first retail store or the second retail store; wherein theupdated virtual shopping list is generated based on a combination of thereceived pricing information and the received inventory information; andproviding the updated virtual shopping list to the user.

In an embodiment, a method for distributing shopping may includeaccessing a virtual shopping list associated with a user, the virtualshopping list identifying a plurality of products. A first indicatorassociated with a prospective visit by the user to a first retail storeis determined. A second indicator associated with a prospective visit bythe user to a second retail store is determined. Pricing information forthe plurality of products located at the first retail store and at thesecond retail store is received. Inventory information for the pluralityof products located at the first retail store and at the second retailstore is received. An updated virtual shopping list is automaticallygenerated, including an indication of a location for purchasing each ofthe plurality of products. The indication of the location is associatedwith the first retail store or the second retail store. The updatedvirtual shopping list is generated based on a combination of thereceived pricing information and the received inventory information. Theupdated virtual shopping list is provided to the user.

In an embodiment, a non-transitory computer-readable medium may includeinstructions that when executed by a processor cause the processor toimplement a frictionless shopping method based on estimated shopperintegrity. The method includes receiving, from at least a first of aplurality of sensors associated with a retail store, first sensed dataindicating a selection of a first product by a customer; determining,from the first sensed data, a first property associated with the firstproduct; issuing a first query to the customer with respect to the firstproperty associated with the first product and, in response, receivingfirst information from the customer; comparing the first informationreceived from the customer with the first property associated with thefirst product; determining, based on the comparison between the firstinformation received from the customer with the first propertyassociated with the first product, an integrity indicator associatedwith the customer; receiving, from at least a second of the plurality ofsensors associated with the retail store, second sensed data indicatinga selection of a second product by the customer; determining, based onthe integrity indicator, at least one action to take in order todetermine a second property associated with the second product; andinitiating the at least one action.

In an embodiment, a frictionless shopping system may include at leastone processor configured with access to instructions that when executedby the at least one processor cause the at least one processor toimplement a frictionless shopping method based on estimated shopperintegrity. The method includes receiving, from at least a first of aplurality of sensors associated with a retail store, first sensed dataindicating a selection of a first product by a customer; determining,from the first sensed data, a first property associated with the firstproduct; issuing a first query to the customer with respect to the firstproperty associated with the first product and, in response, receivingfirst information from the customer; comparing the first informationreceived from the customer with the first property associated with thefirst product; determining, based on the comparison between the firstinformation received from the customer with the first propertyassociated with the first product, an integrity indicator associatedwith the customer; receiving, from at least a second of the plurality ofsensors associated with the retail store, second sensed data indicatinga selection of a second product by the customer; determining, based onthe integrity indicator, at least one action to take in order todetermine a second property associated with the second product; andinitiating the at least one action.

In an embodiment, a frictionless shopping method based on estimatedshopper integrity may include receiving, from at least a first of aplurality of sensors associated with a retail store, first sensed dataindicating a selection of a first product by a customer; determining,from the first sensed data, a first property associated with the firstproduct; issuing a first query to the customer with respect to the firstproperty associated with the first product and, in response, receivingfirst information from the customer; comparing the first informationreceived from the customer with the first property associated with thefirst product; determining, based on the comparison between the firstinformation received from the customer with the first propertyassociated with the first product, an integrity indicator associatedwith the customer; receiving, from at least a second of the plurality ofsensors associated with the retail store, second sensed data indicatinga selection of a second product by the customer; determining, based onthe integrity indicator, at least one action to take in order todetermine a second property associated with the second product; andinitiating the at least one action.

In an embodiment, a non-transitory computer-readable medium may includeinstructions that when executed by a processor cause the processor toperform a method for associating online digital activities withconditions in a retail store. The method includes detecting a pluralityof online digital activities including subject matter related to aretail store, wherein the plurality of online digital activitiesoriginate from customers in the retail store; receiving positioninformation indicative of positions in the retail store from which eachof the detected plurality of online digital activities originated;clustering the detected plurality of online digital activities based onthe received position information to provide one or more clusters ofdetected online digital activities, wherein each of the one or moreclusters is associated with a different region of the retail store;using an image acquisition device to capture one or more imagesrepresentative of a particular region of the retail store associatedwith a particular cluster of detected online digital activities;analyzing the one or more captured images to detect a condition of theretail store associated with the particular cluster of detected onlinedigital activities; and automatically initiating at least one remedialaction to address the detected condition

In an embodiment, a system may associate online digital activities withpositions in retail stores. The system includes at least one processingunit configured to detect a plurality of online digital activitiesincluding subject matter related to a retail store, wherein theplurality of online digital activities originate from customers in theretail store; receive position information indicative of positions inthe retail store from which each of the detected plurality of onlinedigital activities originated; cluster the detected plurality of onlinedigital activities based on the received position information to provideone or more clusters of detected online digital activities, wherein eachof the one or more clusters is associated with a different region of theretail store; use an image acquisition device to capture one or moreimages representative of a particular region of the retail storeassociated with a particular cluster of detected online digitalactivities; analyze the one or more captured images to detect acondition of the retail store associated with the particular cluster ofdetected online digital activities; and automatically initiate at leastone remedial action to address the detected condition.

In an embodiment, a method may associate digital activities withpositions in retail stores. The method includes detecting a plurality ofonline digital activities including subject matter related to a retailstore, wherein the plurality of online digital activities originate fromcustomers in the retail store; receiving position information indicativeof positions in the retail store from which each of the detectedplurality of online digital activities originated; clustering thedetected plurality of online digital activities based on the receivedposition information to provide one or more clusters of detected onlinedigital activities, wherein each of the one or more clusters isassociated with a different region of the retail store; using an imageacquisition device to capture one or more images representative of aparticular region of the retail store associated with a particularcluster of detected online digital activities; analyzing the one or morecaptured images to detect a condition of the retail store associatedwith the particular cluster of detected online digital activities; andautomatically initiating at least one remedial action to address thedetected condition.

In an embodiment, a non-transitory computer-readable medium may includeinstructions that when executed by a processor cause the processor toperform a method for visualization of retail shelves. The methodincludes determining product availability data indicative of anavailability of one or more products on at least one retail shelf in aretail store; using the product availability data to generate asynthetic visual representation of the one or more products on the atleast one retail shelf; presenting the synthetic visual representationon a display; receiving from a user an input indicative of a selectionof at least a portion of the synthetic visual representation; and inresponse to receiving the input, presenting on the display an actualimage corresponding to the selected at least a portion of the syntheticvisual representation, wherein the actual image was acquired with animage sensor.

In an embodiment, a system for visualization of retail shelves mayinclude at least one processor. The at least one processor may beprogrammed to the steps of determining product availability dataindicative of an availability of one or more products on at least oneretail shelf in a retail store; using the product availability data togenerate a synthetic visual representation of the one or more productson the at least one retail shelf; presenting the synthetic visualrepresentation on a display; receiving from a user an input indicativeof a selection of at least a portion of the synthetic visualrepresentation; and in response to receiving the input, presenting onthe display an actual image corresponding to the selected at least aportion of the synthetic visual representation, wherein the actual imagewas acquired with an image sensor.

In an embodiment, a method for visualization of retail shelves mayinclude determining product availability data indicative of anavailability of one or more products on at least one retail shelf in aretail store; using the product availability data to generate asynthetic visual representation of the one or more products on the atleast one retail shelf; presenting the synthetic visual representationon a display; receiving from a user an input indicative of a selectionof at least a portion of the synthetic visual representation; and inresponse to receiving the input, presenting on the display an actualimage corresponding to the selected at least a portion of the syntheticvisual representation, wherein the actual image was acquired with animage sensor.

In an embodiment, a non-transitory computer-readable medium may includeinstructions that when executed by a processor cause the processor toperform a method for facilitating camera system inspection in anextended reality environment. The method includes receiving cameralocation information associated with a plurality of cameras deployed ina retail store, the camera location information being indicative of anactual position of each camera of the plurality of cameras; receiving,for each camera of the plurality of cameras, at least one camera statusindicator; using the received camera location information and thereceived camera status indicators to generate a digital signal; andproviding the generated digital signal to an extended reality device tothereby cause the extended reality device to display a visualrepresentation of the at least one status indicator for each camera ofthe plurality of cameras relative to the actual position of the camerain the retail store.

In an embodiment, a system for facilitating camera system inspection inan extended reality environment may include at least one processor. Theat least one processor may be programmed to receive camera locationinformation associated with a plurality of cameras deployed in a retailstore, the camera location information being indicative of an actualposition of each camera of the plurality of cameras; receive, for eachcamera of the plurality of cameras, at least one camera statusindicator; use the received camera location information and the receivedcamera status indicators to generate a digital signal; and provide thegenerated digital signal to an extended reality device to thereby causethe extended reality device to display a visual representation of the atleast one status indicator for each camera of the plurality of camerasrelative to the actual position of the camera in the retail store.

In an embodiment, a method for facilitating camera system inspection inan extended reality environment may include receiving camera locationinformation associated with a plurality of cameras deployed in a retailstore, the camera location information being indicative of an actualposition of each camera of the plurality of cameras; receiving, for eachcamera of the plurality of cameras, at least one camera statusindicator; using the received camera location information and thereceived camera status indicators to generate a digital signal; andproviding the generated digital signal to an extended reality device tothereby cause the extended reality device to display a visualrepresentation of the at least one status indicator for each camera ofthe plurality of cameras relative to the actual position of the camerain the retail store.

In an embodiment, a non-transitory computer-readable medium may includeinstructions that when executed by at least one processor cause the atleast one processor to perform a method for triggering actions inresponse to point of sales data. The method includes obtaining point ofsale data from a retail store, analyzing the point of sale data toidentify at least one anomalous transaction, information configured tocause capturing of image data from the retail store, analyzing the imagedata relating to the at least one anomalous transaction to determine atleast one condition associated with the at least one anomaloustransaction in the retail store, and based on the analyzed image datarelating to the at least one anomalous transaction, generating anindicator associated with the at least one condition.

In an embodiment, a method for triggering actions in response to salesinformation may include obtaining point of sale data from a retailstore; analyzing the point of sale data to identify at least oneanomalous transaction; in response to the identified at least oneanomalous transaction, providing information configured to causecapturing of image data from the retail store; and analyzing the imagedata relating to the at least one anomalous transaction to determine atleast one condition associated with the at least one anomaloustransaction, generating an indicator associated with the at least onecondition.

In an embodiment, a system for monitoring actions associated with aretail store or retail space may include at least one processorconfigured to execute instructions for performing a method. The methodincludes obtaining point of sale data from a retail store; analyzing thepoint of sale data to identify at least one anomalous transaction; inresponse to the identified at least one anomalous transaction, providinginformation configured to cause capturing of image data from the retailstore; analyzing the image data relating to the at least one anomaloustransaction to determine at least one condition associated with the atleast one anomalous transaction in the retail store; and based on theanalyzed image data relating to the at least one anomalous transaction,generating an indicator associated with the at least one condition.

In an embodiment, a non-transitory computer-readable medium may includeinstructions that when executed by at least one processor cause the atleast one processor to perform a method for automatically maintaining aretail space. The method includes receiving image data captured from aretail store; analyzing the received image data to determine that, at acapturing time, a first product type is out of stock; in response to thedetermination that the first product type is out of stock at thecapturing time, generating first information configured to cause atleast one action in response to the determination that the first producttype is out of stock; receiving point of sales data from the retailstore, wherein the point of sales data indicates a purchase of the firstproduct type occurring after the capturing time, but before initiationof the at least one action; and in response to the received point ofsales data, generating second information configured to cancel the atleast one action.

In an embodiment, a method for automatically maintaining a retail spacemay include receiving image data captured from a retail store; analyzingthe received image data to determine that, at a capturing time, a firstproduct type is out of stock; in response to the determination that thefirst product type is out of stock at the capturing time, generatingfirst information configured to cause at least one action in response tothe determination that the first product type is out of stock; receivingpoint of sales data from the retail store, wherein the point of salesdata indicates a purchase of the first product type occurring after thecapturing time, but before initiation of the at least one action; and inresponse to the received point of sales data, generating secondinformation configured to cancel the at least one action.

In an embodiment, a system for automatically maintaining a retail spacemay include at least one processor configured to execute instructionsfor performing a method. The method includes receiving image datacaptured from a retail store; analyzing the received image data todetermine that, at a capturing time, a first product type is out ofstock; in response to the determination that the first product type isout of stock at the capturing time, generating first informationconfigured to cause at least one action in response to the determinationthat the first product type is out of stock; receiving point of salesdata from the retail store, wherein the point of sales data indicates apurchase of the first product type occurring after the capturing time,but before initiation of the at least one action; and in response to thereceived point of sales data, generating second information configuredto cancel the at least one action.

The foregoing general description and the following detailed descriptionare exemplary and explanatory only and are not restrictive of theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate various disclosed embodiments. Inthe drawings:

FIG. 1 is an illustration of an exemplary system for analyzinginformation collected from a retail store.

FIG. 2 is a block diagram that illustrates some of the components of animage processing system, consistent with the present disclosure.

FIG. 3 is a block diagram that illustrates an exemplary embodiment of acapturing device, consistent with the present disclosure.

FIG. 4A is a schematic illustration of an example configuration forcapturing image data in a retail store, consistent with the presentdisclosure.

FIG. 4B is a schematic illustration of another example configuration forcapturing image data in a retail store, consistent with the presentdisclosure.

FIG. 4C is a schematic illustration of another example configuration forcapturing image data in a retail store, consistent with the presentdisclosure.

FIG. 5A is an illustration of an example system for acquiring images ofproducts in a retail store, consistent with the present disclosure.

FIG. 5B is an illustration of a shelf-mounted camera unit included in afirst housing of the example system of FIG. 5A, consistent with thepresent disclosure.

FIG. 5C is an exploded view illustration of a processing unit includedin a second housing of the example system of FIG. 5A, consistent withthe present disclosure.

FIG. 6A is a top view representation of an aisle in a retail store withmultiple image acquisition systems deployed thereon for acquiring imagesof products, consistent with the present disclosure.

FIG. 6B is a perspective view representation of part of a retailshelving unit with multiple image acquisition systems deployed thereonfor acquiring images of products, consistent with the presentdisclosure.

FIG. 6C provides a diagrammatic representation of how the exemplarydisclosed image acquisition systems may be positioned relative to retailshelving to acquire product images, consistent with the presentdisclosure.

FIG. 7A provides a flowchart of an exemplary method for acquiring imagesof products in retail store, consistent with the present disclosure.

FIG. 7B provides a flowchart of a method for acquiring images ofproducts in retail store, consistent with the present disclosure.

FIG. 8A is a schematic illustration of an example configuration fordetecting products and empty spaces on a store shelf, consistent withthe present disclosure;

FIG. 8B is a schematic illustration of another example configuration fordetecting products and empty spaces on a store shelf, consistent withthe present disclosure;

FIG. 9 is a schematic illustration of example configurations fordetection elements on store shelves, consistent with the presentdisclosure;

FIG. 10A illustrates an exemplary method for monitoring planogramcompliance on a store shelf, consistent with the present disclosure;

FIG. 10B is illustrates an exemplary method for triggering imageacquisition based on product events on a store shelf, consistent withthe present disclosure;

FIG. 11A is a schematic illustration of an example output for a marketresearch entity associated with the retail store, consistent with thepresent disclosure;

FIG. 11B is a schematic illustration of an example output for a supplierof the retail store, consistent with the present disclosure;

FIG. 11C is a schematic illustration of an example output for a managerof the retail store, consistent with the present disclosure;

FIG. 11D is a schematic illustration of two examples outputs for a storeassociate of the retail store, consistent with the present disclosure;and

FIG. 11E is a schematic illustration of an example output for an onlinecustomer of the retail store, consistent with the present disclosure.

FIG. 12 is an illustration of an exemplary retail store product locatorsystem.

FIG. 13A is an illustration of a display screen for a head-worn extendedreality appliance as described herein.

FIG. 13B is an illustration of a hand-held mobile communicationsembodiment of an extended reality appliance with a see-through screen asdescribed herein.

FIG. 13C is an illustration of a display screen of the extended realityappliance of FIG. 13B with a graphic overlay having visual indicators asdescribed herein.

FIG. 14A is a flowchart representing an exemplary method for locating anavailable alternative product in a store, consistent with disclosedembodiments.

FIG. 14B is a flowchart representing an exemplary method for locating anavailable alternative product in a store, consistent with disclosedembodiments.

FIG. 15A is an illustration of a display screen of a wearable extendedreality appliance showing a shopper notification as described herein.

FIG. 15B is an illustration of a shopper with a wearable extendedreality appliance as described herein.

FIG. 15C is an illustration of an in-store infrastructure display screenmounted on a shelf-space for displaying a shopper notification asdescribed herein.

FIG. 16A is an illustration of a display screen of the mobile device ofFIG. 12 displaying a shopper notification as described herein.

FIG. 16B is an illustration of a display screen of the mobile device ofFIG. 12 displaying a shopper notification as described herein.

FIG. 17 is a flowchart representing an exemplary method for providing anin-store shopper notification about the online ordering of anunavailable in-store product, consistent with disclosed embodiments.

FIG. 18 is an example map showing possible routes for a user to take tovisit a first retail store and a second retail store, consistent withthe present disclosure.

FIG. 19A is a schematic illustration of a shopping list, consistent withthe present disclosure.

FIG. 19B is a schematic illustration of an exemplary output of a methodfor distributed shopping, consistent with the present disclosure.

FIG. 19C is a schematic illustration of a second exemplary output of amethod for distributed shopping, consistent with the present disclosure.

FIG. 19D is a schematic illustration of a third exemplary output of amethod for distributed shopping, consistent with the present disclosure.

FIG. 20 is a flowchart of an exemplary method for distributed shopping,consistent with the present disclosure.

FIG. 21A is an exemplary screenshot of a customer query displayed on adisplay attached to shopping cart, consistent with the presentdisclosure.

FIG. 21B is a top plan view of a shopping cart, including one or moresensors and a display screen attached to the shopping cart, consistentwith the present disclosure.

FIG. 22A is an exemplary screenshot of a customer query displayed on thecustomer's mobile device, consistent with the present disclosure.

FIG. 22B is a schematic illustration of a customer taking a picture of aproduct with the customer's mobile device, consistent with the presentdisclosure.

FIG. 23 is a flowchart of an exemplary method for frictionless shoppingbased on estimated shopper integrity, consistent with the presentdisclosure.

FIGS. 24A and 24B illustrate an exemplary implementation of a system forassociating online digital activities with conditions in a retail store,consistent with the present disclosure.

FIG. 25 illustrates an exemplary implementation of a listing of onlinedigital activities performed via multiple digital platforms, consistentwith the present disclosure.

FIG. 26 is a flowchart of an example process for associating digitalactivities with positions in retail stores, consistent with the presentdisclosure.

FIG. 27A illustrates an example hybrid visualization interface that maybe displayed to a user, consistent with the present disclosure.

FIG. 27B illustrates an example hybrid visualization interfacedisplaying an actual image associated with a product, consistent withthe present disclosure.

FIG. 28A illustrates an example hybrid visualization interfacedisplaying an actual image associated with a shelf, consistent with thepresent disclosure.

FIG. 28B illustrates an example hybrid visualization interfacedisplaying a plurality of actual images associated with a product,consistent with the present disclosure.

FIG. 29 is a flowchart of an exemplary process for visualization ofretail shelves, consistent with the present disclosure.

FIG. 30A illustrates an example view of a retail store that may beenhanced by an extended reality device, consistent with the disclosedembodiments.

FIGS. 30B, 30C, and 30D illustrate example information that may bedisplayed within an extended reality view using an extended realitydevice, consistent with the disclosed embodiments.

FIGS. 31A, 31B, 31C, and 31D illustrate example visual representationsof a field of view of one or more cameras in an extended realityenvironment, consistent with the disclosed embodiments.

FIG. 32 is a flowchart of an exemplary process for facilitating camerasystem inspection in an extended reality environment, consistent withthe present disclosure.

FIG. 33 is an example of an anomalous transaction processed by imageanalysis, consistent with the present disclosure.

FIG. 34 is an example of information generated to notify a user ofanomalous transactions, consistent with the present disclosure.

FIG. 35 is a flowchart of a method for processing anomaloustransactions, consistent with the present disclosure.

FIG. 36 is an example image including a representation of a shelvingunit, consistent with the present disclosure.

FIG. 37 is example of using point of sale data to cancel an actionrelated to an out of stock condition in a retail store, consistent withthe present disclosure.

FIG. 38 is a flowchart of a method for automatically maintaining aretail space, consistent with the present disclosure.

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar parts.While several illustrative embodiments are described herein,modifications, adaptations and other implementations are possible. Forexample, substitutions, additions, or modifications may be made to thecomponents illustrated in the drawings, and the illustrative methodsdescribed herein may be modified by substituting, reordering, removing,or adding steps to the disclosed methods. Accordingly, the followingdetailed description is not limited to the disclosed embodiments andexamples. Instead, the proper scope is defined by the appended claims.

The present disclosure is directed to systems and methods for processingimages captured in a retail store. As used herein, the term “retailstore” or simply “store” refers to an establishment offering productsfor sale by direct selection by customers physically or virtuallyshopping within the establishment. The retail store may be anestablishment operated by a single retailer (e.g., supermarket) or anestablishment that includes stores operated by multiple retailers (e.g.,a shopping mall). Embodiments of the present disclosure includereceiving an image depicting a store shelf having at least one productdisplayed thereon. As used herein, the term “store shelf” or simply“shelf” refers to any suitable physical structure which may be used fordisplaying products in a retail environment. In one embodiment the storeshelf may be part of a shelving unit including a number of individualstore shelves. In another embodiment, the store shelf may include adisplay unit having a single-level or multi-level surfaces.

Consistent with the present disclosure, the system may process imagesand image data acquired by a capturing device to determine informationassociated with products displayed in the retail store. The term“capturing device” refers to any device configured to acquire image datarepresentative of products displayed in the retail store. Examples ofcapturing devices may include a digital camera, a time-of-flight camera,a stereo camera, an active stereo camera, a depth camera, a Lidarsystem, a laser scanner, CCD based devices, or any other sensor basedsystem capable of converting received light into electric signals. Theterm “image data” refers to any form of data generated based on opticalsignals in the near-infrared, infrared, visible, and ultravioletspectrums (or any other suitable radiation frequency range). Consistentwith the present disclosure, the image data may include pixel datastreams, digital images, digital video streams, data derived fromcaptured images, and data that may be used to construct a 3D image. Theimage data acquired by a capturing device may be transmitted by wired orwireless transmission to a remote server. In one embodiment, thecapturing device may include a stationary camera with communicationlayers (e.g., a dedicated camera fixed to a store shelf, a securitycamera, etc.). Such an embodiment is described in greater detail belowwith reference to FIG. 4A. In another embodiment, the capturing devicemay include a handheld device (e.g., a smartphone, a tablet, a mobilestation, a personal digital assistant, a laptop, and more) or a wearabledevice (e.g., smart glasses, a smartwatch, a clip-on camera). Such anembodiment is described in greater detail below with reference to FIG.4B. In another embodiment, the capturing device may include a roboticdevice with one or more cameras operated remotely or autonomously (e.g.,an autonomous robotic device, a drone, a robot on a track, and more).Such an embodiment is described in greater detail below with referenceto FIG. 4C.

In some embodiments, the capturing device may include one or more imagesensors. The term “image sensor” refers to a device capable of detectingand converting optical signals in the near-infrared, infrared, visible,and ultraviolet spectrums into electrical signals. The electricalsignals may be used to form image data (e.g., an image or a videostream) based on the detected signal. Examples of image sensors mayinclude semiconductor charge-coupled devices (CCD), active pixel sensorsin complementary metal-oxide-semiconductor (CMOS), or N-typemetal-oxide-semiconductors (NMOS, Live MOS). In some cases, the imagesensor may be part of a camera included in the capturing device.

Embodiments of the present disclosure further include analyzing imagesto detect and identify different products. As used herein, the term“detecting a product” may broadly refer to determining an existence ofthe product. For example, the system may determine the existence of aplurality of distinct products displayed on a store shelf. By detectingthe plurality of products, the system may acquire different detailsrelative to the plurality of products (e.g., how many products on astore shelf are associated with a same product type), but it does notnecessarily gain knowledge of the type of product. In contrast, the term“identifying a product” may refer to determining a unique identifierassociated with a specific type of product that allows inventorymanagers to uniquely refer to each product type in a product catalogue.Additionally or alternatively, the term “identifying a product” mayrefer to determining a unique identifier associated with a specificbrand of products that allows inventory managers to uniquely refer toproducts, e.g., based on a specific brand in a product catalog.Additionally or alternatively, the term “identifying a product” mayrefer to determining a unique identifier associated with a specificcategory of products that allows inventory managers to uniquely refer toproducts, e.g., based on a specific category in a product catalog. Insome embodiments, the identification may be made based at least in parton visual characteristics of the product (e.g., size, shape, logo, text,color, etc.). The unique identifier may include any codes that may beused to search a catalog, such as a series of digits, letters, symbols,or any combinations of digits, letters, and symbols. Consistent with thepresent disclosure, the terms “determining a type of a product” and“determining a product type” may also be used interchangeably in thisdisclosure with reference to the term “identifying a product.”

Embodiments of the present disclosure further include determining atleast one characteristic of the product for determining the type of theproduct. As used herein, the term “characteristic of the product” refersto one or more visually discernable features attributed to the product.Consistent with the present disclosure, the characteristic of theproduct may assist in classifying and identifying the product. Forexample, the characteristic of the product may be associated with theornamental design of the product, the size of the product, the shape ofthe product, the colors of the product, the brand of the product, a logoor text associated with the product (e.g., on a product label), andmore. In addition, embodiments of the present disclosure further includedetermining a confidence level associated with the determined type ofthe product. The term “confidence level” refers to any indication,numeric or otherwise, of a level (e.g., within a predetermined range)indicative of an amount of confidence the system has that the determinedtype of the product is the actual type of the product. For example, theconfidence level may have a value between 1 and 10, alternatively, theconfidence level may be expressed as a percentage.

In some cases, the system may compare the confidence level to athreshold. The term “threshold” as used herein denotes a referencevalue, a level, a point, or a range of values, for which, when theconfidence level is above it (or below it depending on a particular usecase), the system may follow a first course of action and, when theconfidence level is below it (or above it depending on a particular usecase), the system may follow a second course of action. The value of thethreshold may be predetermined for each type of product or may bedynamically selected based on different considerations. In oneembodiment, when the confidence level associated with a certain productis below a threshold, the system may obtain contextual information toincrease the confidence level. As used herein, the term “contextualinformation” (or “context”) refers to any information having a direct orindirect relationship with a product displayed on a store shelf. In someembodiments, the system may retrieve different types of contextualinformation from captured image data and/or from other data sources. Insome cases, contextual information may include recognized types ofproducts adjacent to the product under examination. In other cases,contextual information may include text appearing on the product,especially where that text may be recognized (e.g., via OCR) andassociated with a particular meaning. Other examples of types ofcontextual information may include logos appearing on the product, alocation of the product in the retail store, a brand name of theproduct, a price of the product, product information collected frommultiple retail stores, product information retrieved from a catalogassociated with a retail store, etc.

In some embodiments, machine learning algorithms (also referred to asmachine learning models in the present disclosure) may be trained usingtraining examples such as described below. Some non-limiting examples ofsuch machine learning algorithms may include classification algorithms,data regressions algorithms, image segmentation algorithms, visualdetection algorithms (such as object detectors, face detectors, persondetectors, motion detectors, edge detectors, etc.), visual recognitionalgorithms (such as face recognition, person recognition, objectrecognition, etc.), speech recognition algorithms, mathematicalembedding algorithms, natural language processing algorithms, supportvector machines, random forests, nearest neighbors algorithms, deeplearning algorithms, artificial neural network algorithms, convolutionalneural network algorithms, recurrent neural network algorithms, linearmachine learning models, non-linear machine learning models, ensemblealgorithms, and so forth. For example, a trained machine learningalgorithm may comprise an inference model, such as a predictive model, aclassification model, a data regression model, a clustering model, asegmentation model, an artificial neural network (such as a deep neuralnetwork, a convolutional neural network, a recurrent neural network,etc.), a random forest, a support vector machine, and so forth. In someexamples, the training examples may include example inputs together withthe desired outputs corresponding to the example inputs. Further, insome examples, training machine learning algorithms using the trainingexamples may generate a trained machine learning algorithm, and thetrained machine learning algorithm may be used to estimate outputs forinputs not included in the training examples. In some examples,engineers, scientists, processes, and machines that train machinelearning algorithms may further use validation examples and/or testexamples. For example, validation examples and/or test examples mayinclude example inputs together with the desired outputs correspondingto the example inputs, a trained machine learning algorithm and/or anintermediately trained machine learning algorithm may be used toestimate outputs for the example inputs of the validation examplesand/or test examples, the estimated outputs may be compared to thecorresponding desired outputs, and the trained machine learningalgorithm and/or the intermediately trained machine learning algorithmmay be evaluated based on a result of the comparison. In some examples,a machine learning algorithm may have parameters and hyper parameters,where the hyper parameters may be set manually by a person orautomatically by an process external to the machine learning algorithm(such as a hyper parameter search algorithm), and the parameters of themachine learning algorithm may be set by the machine learning algorithmbased on the training examples. In some implementations, thehyper-parameters may be set based on the training examples and thevalidation examples, and the parameters may be set based on the trainingexamples and the selected hyper-parameters. For example, given thehyper-parameters, the parameters may be conditionally independent of thevalidation examples.

In some embodiments, trained machine learning algorithms (also referredto as machine learning models and trained machine learning models in thepresent disclosure) may be used to analyze inputs and generate outputs,for example, as described below. In some examples, a trained machinelearning algorithm may be used as an inference model that when providedwith an input generates an inferred output. For example, a trainedmachine learning algorithm may include a classification algorithm, theinput may include a sample, and the inferred output may include aclassification of the sample (such as an inferred label, an inferredtag, and so forth). In another example, a trained machine learningalgorithm may include a regression model, the input may include asample, and the inferred output may include an inferred valuecorresponding to the sample. In yet another example, a trained machinelearning algorithm may include a clustering model, the input may includea sample, and the inferred output may include an assignment of thesample to at least one cluster. In an additional example, a trainedmachine learning algorithm may include a classification algorithm, theinput may include an image, and the inferred output may include aclassification of an item depicted in the image. In yet another example,a trained machine learning algorithm may include a regression model, theinput may include an image, and the inferred output may include aninferred value corresponding to an item depicted in the image (such asan estimated property of the item, such as size, volume, age of a persondepicted in the image, distance from an item depicted in the image, andso forth). In an additional example, a trained machine learningalgorithm may include an image segmentation model, the input may includean image, and the inferred output may include a segmentation of theimage. In yet another example, a trained machine learning algorithm mayinclude an object detector, the input may include an image, and theinferred output may include one or more detected objects in the imageand/or one or more locations of objects within the image. In someexamples, the trained machine learning algorithm may include one or moreformulas and/or one or more functions and/or one or more rules and/orone or more procedures, the input may be used as input to the formulasand/or functions and/or rules and/or procedures, and the inferred outputmay be based on the outputs of the formulas and/or functions and/orrules and/or procedures (for example, selecting one of the outputs ofthe formulas and/or functions and/or rules and/or procedures, using astatistical measure of the outputs of the formulas and/or functionsand/or rules and/or procedures, and so forth).

In some embodiments, analyzing image data (for example by the methods,steps and modules described herein) may comprise analyzing the imagedata to obtain a preprocessed image data, and subsequently analyzing theimage data and/or the preprocessed image data to obtain the desiredoutcome. One of ordinary skill in the art will recognize that thefollowings are examples, and that the image data may be preprocessedusing other kinds of preprocessing methods. In some examples, the imagedata may be preprocessed by transforming the image data using atransformation function to obtain a transformed image data, and thepreprocessed image data may comprise the transformed image data. Forexample, the transformed image data may comprise one or moreconvolutions of the image data. For example, the transformation functionmay comprise one or more image filters, such as low-pass filters,high-pass filters, band-pass filters, all-pass filters, and so forth. Insome examples, the transformation function may comprise a nonlinearfunction. In some examples, the image data may be preprocessed bysmoothing at least parts of the image data, for example using Gaussianconvolution, using a median filter, and so forth. In some examples, theimage data may be preprocessed to obtain a different representation ofthe image data. For example, the preprocessed image data may comprise: arepresentation of at least part of the image data in a frequency domain;a Discrete Fourier Transform of at least part of the image data; aDiscrete Wavelet Transform of at least part of the image data; atime/frequency representation of at least part of the image data; arepresentation of at least part of the image data in a lower dimension;a lossy representation of at least part of the image data; a losslessrepresentation of at least part of the image data; a time ordered seriesof any of the above; any combination of the above; and so forth. In someexamples, the image data may be preprocessed to extract edges, and thepreprocessed image data may comprise information based on and/or relatedto the extracted edges. In some examples, the image data may bepreprocessed to extract image features from the image data. Somenon-limiting examples of such image features may comprise informationbased on and/or related to: edges; corners; blobs; ridges; ScaleInvariant Feature Transform (SIFT) features; temporal features; and soforth. In some examples, analyzing the image data may includecalculating at least one convolution of at least a portion of the imagedata, and using the calculated at least one convolution to calculate atleast one resulting value and/or to make determinations,identifications, recognitions, classifications, and so forth.

In some embodiments, analyzing image data (for example by the methods,steps and modules described herein) may comprise analyzing the imagedata and/or the preprocessed image data using one or more rules,functions, procedures, artificial neural networks, object detectionalgorithms, face detection algorithms, visual event detectionalgorithms, action detection algorithms, motion detection algorithms,background subtraction algorithms, inference models, and so forth. Somenon-limiting examples of such inference models may include: an inferencemodel preprogrammed manually; a classification model; a regressionmodel; a result of training algorithms, such as machine learningalgorithms and/or deep learning algorithms, on training examples, wherethe training examples may include examples of data instances, and insome cases, a data instance may be labeled with a corresponding desiredlabel and/or result; and so forth. In some embodiments, analyzing imagedata (for example by the methods, steps and modules described herein)may comprise analyzing pixels, voxels, point cloud, range data, etc.included in the image data.

A convolution may include a convolution of any dimension. Aone-dimensional convolution is a function that transforms an originalsequence of numbers to a transformed sequence of numbers. Theone-dimensional convolution may be defined by a sequence of scalars.Each particular value in the transformed sequence of numbers may bedetermined by calculating a linear combination of values in asubsequence of the original sequence of numbers corresponding to theparticular value. A result value of a calculated convolution may includeany value in the transformed sequence of numbers. Likewise, ann-dimensional convolution is a function that transforms an originaln-dimensional array to a transformed array. The n-dimensionalconvolution may be defined by an n-dimensional array of scalars (knownas the kernel of the n-dimensional convolution). Each particular valuein the transformed array may be determined by calculating a linearcombination of values in an n-dimensional region of the original arraycorresponding to the particular value. A result value of a calculatedconvolution may include any value in the transformed array. In someexamples, an image may comprise one or more components (such as colorcomponents, depth component, etc.), and each component may include a twodimensional array of pixel values. In one example, calculating aconvolution of an image may include calculating a two-dimensionalconvolution on one or more components of the image. In another example,calculating a convolution of an image may include stacking arrays fromdifferent components to create a three-dimensional array, andcalculating a three dimensional convolution on the resulting threedimensional array. In some examples, a video may comprise one or morecomponents (such as color components, depth component, etc.), and eachcomponent may include a three-dimensional array of pixel values (withtwo spatial axes and one temporal axis). In one example, calculating aconvolution of a video may include calculating a three dimensionalconvolution on one or more components of the video. In another example,calculating a convolution of a video may include stacking arrays fromdifferent components to create a four dimensional array, and calculatinga four dimensional convolution on the resulting four dimensional array.

Reference is now made to FIG. 1 , which shows an example of a system 100for analyzing information collected from retail stores 105 (for example,retail store 105A, retail store 105B, and retail store 105C). In oneembodiment, system 100 may represent a computer-based system that mayinclude computer system components, desktop computers, workstations,tablets, handheld computing devices, memory devices, and/or internalnetwork(s) connecting the components. System 100 may include or beconnected to various network computing resources (e.g., servers,routers, switches, network connections, storage devices, etc.) necessaryto support the services provided by system 100. In one embodiment,system 100 may enable identification of products in retail stores 105based on analysis of captured images. In another embodiment, system 100may enable a supply of information based on analysis of captured imagesto a market research entity 110 and to different suppliers 115 of theidentified products in retail stores 105 (for example, supplier 115A,supplier 115B, and supplier 115C). In another embodiment, system 100 maycommunicate with a user 120 (sometimes referred to herein as a customer,but which may include individuals associated with a retail environmentother than customers, such as store associate, data collection agent,etc.) about different products in retail stores 105. In one example,system 100 may receive images of products captured by user 120. Inanother example, system 100 may provide information to user 120determined based on automatic machine analysis of images captured by oneor more capturing devices 125 associated with retail stores 105.

System 100 may also include an image processing unit 130 to execute theanalysis of images captured by the one or more capturing devices 125.Image processing unit 130 may include a server 135 operatively connectedto a database 140. Image processing unit 130 may include one or moreservers connected by a communication network, a cloud platform, and soforth. Consistent with the present disclosure, image processing unit 130may receive raw or processed data from capturing device 125 viarespective communication links, and provide information to differentsystem components using a network 150. Specifically, image processingunit 130 may use any suitable image analysis technique including, forexample, object recognition, object detection, image segmentation,feature extraction, optical character recognition (OCR), object-basedimage analysis, shape region techniques, edge detection techniques,pixel-based detection, artificial neural networks, convolutional neuralnetworks, etc. In addition, image processing unit 130 may useclassification algorithms to distinguish between the different productsin the retail store. In some embodiments, image processing unit 130 mayutilize suitably trained machine learning algorithms and models toperform the product identification. Network 150 may facilitatecommunications and data exchange between different system componentswhen these components are coupled to network 150 to enable output ofdata derived from the images captured by the one or more capturingdevices 125. In some examples, the types of outputs that imageprocessing unit 130 may generate may include identification of products,indicators of product quantity, indicators of planogram compliance,indicators of service-improvement events (e.g., a cleaning event, arestocking event, a rearrangement event, etc.), and various reportsindicative of the performances of retail stores 105. Additional examplesof the different outputs enabled by image processing unit 130 aredescribed below with reference to FIGS. 11A-11E and throughout thedisclosure.

Consistent with the present disclosure, network 150 may be any type ofnetwork (including infrastructure) that provides communications,exchanges information, and/or facilitates the exchange of informationbetween the components of system 100. For example, network 150 mayinclude or be part of the Internet, a Local Area Network, wirelessnetwork (e.g., a Wi-Fi/802.11 network), or other suitable connections.In other embodiments, one or more components of system 100 maycommunicate directly through dedicated communication links, such as, forexample, a telephone network, an extranet, an intranet, the Internet,satellite communications, off-line communications, wirelesscommunications, transponder communications, a local area network (LAN),a wide area network (WAN), a virtual private network (VPN), and soforth.

In one example configuration, server 135 may be a cloud server thatprocesses images received directly (or indirectly) from one or morecapturing devices 125 and processes the images to detect and/or identifyat least some of the plurality of products in the image based on visualcharacteristics of the plurality of products. The term “cloud server”refers to a computer platform that provides services via a network, suchas the Internet. In this example configuration, server 135 may usevirtual machines that may not correspond to individual hardware. Forexample, computational and/or storage capabilities may be implemented byallocating appropriate portions of desirable computation/storage powerfrom a scalable repository, such as a data center or a distributedcomputing environment. In one example, server 135 may implement themethods described herein using customized hard-wired logic, one or moreApplication Specific Integrated Circuits (ASICs) or Field ProgrammableGate Arrays (FPGAs), firmware, and/or program logic which, incombination with the computer system, cause server 135 to be aspecial-purpose machine.

In another example configuration, server 135 may be part of a systemassociated with a retail store that communicates with capturing device125 using a wireless local area network (WLAN) and may provide similarfunctionality as a cloud server. In this example configuration, server135 may communicate with an associated cloud server (not shown) andcloud database (not shown). The communications between the store serverand the cloud server may be used in a quality enforcement process, forupgrading the recognition engine and the software from time to time, forextracting information from the store level to other data users, and soforth. Consistent with another embodiment, the communications betweenthe store server and the cloud server may be discontinuous (purposely orunintentional) and the store server may be configured to operateindependently from the cloud server. For example, the store server maybe configured to generate a record indicative of changes in productplacement that occurred when there was a limited connection (or noconnection) between the store server and the cloud server, and toforward the record to the cloud server once connection is reestablished.

As depicted in FIG. 1 , server 135 may be coupled to one or morephysical or virtual storage devices such as database 140. Server 135 mayaccess database 140 to detect and/or identify products. The detectionmay occur through analysis of features in the image using an algorithmand stored data. The identification may occur through analysis ofproduct features in the image according to stored product models.Consistent with the present embodiment, the term “product model” refersto any type of algorithm or stored product data that a processor mayaccess or execute to enable the identification of a particular productassociated with the product model. For example, the product model mayinclude a description of visual and contextual properties of theparticular product (e.g., the shape, the size, the colors, the texture,the brand name, the price, the logo, text appearing on the particularproduct, the shelf associated with the particular product, adjacentproducts in a planogram, the location within the retail store, etc.). Insome embodiments, a single product model may be used by server 135 toidentify more than one type of products, such as, when two or moreproduct models are used in combination to enable identification of aproduct. For example, in some cases, a first product model may be usedby server 135 to identify a product category (such models may apply tomultiple product types, e.g., shampoo, soft drinks, etc.), and a secondproduct model may be used by server 135 to identify the product type,product identity, or other characteristics associated with a product. Insome cases, such product models may be applied together (e.g., inseries, in parallel, in a cascade fashion, in a decision tree fashion,etc.) to reach a product identification. In other embodiments, a singleproduct model may be used by server 135 to identify a particular producttype (e.g., 6-pack of 16 ounce Coca-Cola Zero).

Database 140 may be included on a volatile or non-volatile, magnetic,semiconductor, tape, optical, removable, non-removable, or other type ofstorage device or tangible or non-transitory computer-readable medium.Database 140 may also be part of server 135 or separate from server 135.When database 140 is not part of server 135, server 135 may exchangedata with database 140 via a communication link. Database 140 mayinclude one or more memory devices that store data and instructions usedto perform one or more features of the disclosed embodiments. In oneembodiment, database 140 may include any suitable databases, rangingfrom small databases hosted on a work station to large databasesdistributed among data centers. Database 140 may also include anycombination of one or more databases controlled by memory controllerdevices (e.g., server(s), etc.) or software. For example, database 140may include document management systems, Microsoft SQL databases,SharePoint databases, Oracle™ databases, Sybase™ databases, otherrelational databases, or non-relational databases, such as MongoDB andothers.

Consistent with the present disclosure, image processing unit 130 maycommunicate with output devices 145 to present information derived basedon processing of image data acquired by capturing devices 125. The term“output device” is intended to include all possible types of devicescapable of outputting information from server 135 to users or othercomputer systems (e.g., a display screen, a speaker, a desktop computer,a laptop computer, mobile device, tablet, a PDA, etc.), such as 145A,145B, 145C, and 145D. In one embodiment each of the different systemcomponents (i.e., retail stores 105, market research entity 110,suppliers 115, and users 120) may be associated with an output device145, and each system component may be configured to present differentinformation on the output device 145. In one example, server 135 mayanalyze acquired images including representations of shelf spaces. Basedon this analysis, server 135 may compare shelf spaces associated withdifferent products, and output device 145A may present market researchentity 110 with information about the shelf spaces associated withdifferent products. The shelf spaces may also be compared with salesdata, expired products data, and more. Consistent with the presentdisclosure, market research entity 110 may be a part of (or may workwith) supplier 115. In another example, server 135 may determine productcompliance to a predetermined planogram, and output device 145B maypresent to supplier 115 information about the level of productcompliance at one or more retail stores 105 (for example in a specificretail store 105, in a group of retail stores 105 associated withsupplier 115, in all retail stores 105, and so forth). The predeterminedplanogram may be associated with contractual obligations and/or otherpreferences related to the retailer methodology for placement ofproducts on the store shelves. In another example, server 135 maydetermine that a specific store shelf has a type of fault in the productplacement, and output device 145C may present to a manager of retailstore 105 a user-notification that may include information about acorrect display location of a misplaced product, information about astore shelf associated with the misplaced product, information about atype of the misplaced product, and/or a visual depiction of themisplaced product. In another example, server 135 may identify whichproducts are available on the shelf and output device 145D may presentto user 120 an updated list of products.

The components and arrangements shown in FIG. 1 are not intended tolimit the disclosed embodiments, as the system components used toimplement the disclosed processes and features may vary. In oneembodiment, system 100 may include multiple servers 135, and each server135 may host a certain type of service. For example, a first server mayprocess images received from capturing devices 125 to identify at leastsome of the plurality of products in the image, and a second server maydetermine from the identified products in retail stores 105 compliancewith contractual obligations between retail stores 105 and suppliers115. In another embodiment, system 100 may include multiple servers 135,a first type of servers 135 that may process information from specificcapturing devices 125 (e.g., handheld devices of data collection agents)or from specific retail stores 105 (e.g., a server dedicated to aspecific retail store 105 may be placed in or near the store). System100 may further include a second type of servers 135 that collect andprocess information from the first type of servers 135.

FIG. 2 is a block diagram representative of an example configuration ofserver 135. In one embodiment, server 135 may include a bus 200 (or anyother communication mechanism) that interconnects subsystems andcomponents for transferring information within server 135. For example,bus 200 may interconnect a processing device 202, a memory interface204, a network interface 206, and a peripherals interface 208 connectedto an I/O system 210.

Processing device 202, shown in FIG. 2 , may include at least oneprocessor configured to execute computer programs, applications,methods, processes, or other software to execute particular instructionsassociated with embodiments described in the present disclosure. Theterm “processing device” refers to any physical device having anelectric circuit that performs a logic operation. For example,processing device 202 may include one or more processors, integratedcircuits, microchips, microcontrollers, microprocessors, all or part ofa central processing unit (CPU), graphics processing unit (GPU), digitalsignal processor (DSP), field programmable gate array (FPGA), or othercircuits suitable for executing instructions or performing logicoperations. Processing device 202 may include at least one processorconfigured to perform functions of the disclosed methods such as amicroprocessor manufactured by Intel™, Nvidia™, AMD™, and so forth.Processing device 202 may include a single core or multiple coreprocessors executing parallel processes simultaneously. In one example,processing device 202 may be a single core processor configured withvirtual processing technologies. Processing device 202 may implementvirtual machine technologies or other technologies to provide theability to execute, control, run, manipulate, store, etc., multiplesoftware processes, applications, programs, etc. In another example,processing device 202 may include a multiple-core processor arrangement(e.g., dual core, quad core, etc.) configured to provide parallelprocessing functionalities to allow a device associated with processingdevice 202 to execute multiple processes simultaneously. It isappreciated that other types of processor arrangements could beimplemented to provide the capabilities disclosed herein.

Consistent with the present disclosure, the methods and processesdisclosed herein may be performed by server 135 as a result ofprocessing device 202 executing one or more sequences of one or moreinstructions contained in a non-transitory computer-readable storagemedium. As used herein, a non-transitory computer-readable storagemedium refers to any type of physical memory on which information ordata readable by at least one processor may be stored. Examples includerandom access memory (RAM), read-only memory (ROM), volatile memory,nonvolatile memory, hard drives, CD ROMs, DVDs, flash drives, disks, anyother optical data storage medium, any physical medium with patterns ofholes, a RAM, a PROM, an EPROM, a FLASH-EPROM or any other flash memory,NVRAM, a cache, a register, any other memory chip or cartridge, andnetworked versions of the same. The terms “memory” and“computer-readable storage medium” may refer to multiple structures,such as a plurality of memories or computer-readable storage mediumslocated within server 135, or at a remote location. Additionally, one ormore computer-readable storage mediums may be utilized in implementing acomputer-implemented method. The term “computer-readable storage medium”should be understood to include tangible items and exclude carrier wavesand transient signals.

According to one embodiment, server 135 may include network interface206 (which may also be any communications interface) coupled to bus 200.Network interface 206 may provide one-way or two-way data communicationto a local network, such as network 150. Network interface 206 mayinclude an integrated services digital network (ISDN) card, cable modem,satellite modem, or a modem to provide a data communication connectionto a corresponding type of telephone line. As another example, networkinterface 206 may include a local area network (LAN) card to provide adata communication connection to a compatible LAN. In anotherembodiment, network interface 206 may include an Ethernet port connectedto radio frequency receivers and transmitters and/or optical (e.g.,infrared) receivers and transmitters. The specific design andimplementation of network interface 206 depends on the communicationsnetwork(s) over which server 135 is intended to operate. As describedabove, server 135 may be a cloud server or a local server associatedwith retail store 105. In any such implementation, network interface 206may be configured to send and receive electrical, electromagnetic, oroptical signals, through wires or wirelessly, that may carry analog ordigital data streams representing various types of information. Inanother example, the implementation of network interface 206 may besimilar or identical to the implementation described below for networkinterface 306.

Server 135 may also include peripherals interface 208 coupled to bus200. Peripherals interface 208 may be connected to sensors, devices, andsubsystems to facilitate multiple functionalities. In one embodiment,peripherals interface 208 may be connected to I/O system 210 configuredto receive signals or input from devices and provide signals or outputto one or more devices that allow data to be received and/or transmittedby server 135. In one embodiment I/O system 210 may include or beassociated with output device 145. For example, I/O system 210 mayinclude a touch screen controller 212, an audio controller 214, and/orother input controller(s) 216. Touch screen controller 212 may becoupled to a touch screen 218. Touch screen 218 and touch screencontroller 212 can, for example, detect contact, movement, or breakthereof using any of a plurality of touch sensitivity technologies,including but not limited to capacitive, resistive, infrared, andsurface acoustic wave technologies as well as other proximity sensorarrays or other elements for determining one or more points of contactwith touch screen 218. Touch screen 218 may also, for example, be usedto implement virtual or soft buttons and/or a keyboard. In addition toor instead of touch screen 218, I/O system 210 may include a displayscreen (e.g., CRT, LCD, etc.), virtual reality device, augmented realitydevice, mixed reality device, and so forth. Specifically, touch screencontroller 212 (or display screen controller) and touch screen 218 (orany of the alternatives mentioned above) may facilitate visual outputfrom server 135. Audio controller 214 may be coupled to a microphone 220and a speaker 222 to facilitate voice-enabled functions, such as voicerecognition, voice replication, digital recording, and telephonyfunctions. Specifically, audio controller 214 and speaker 222 mayfacilitate audio output from server 135. The other input controller(s)216 may be coupled to other input/control devices 224, such as one ormore buttons, keyboards, rocker switches, thumb-wheel, infrared port,USB port, image sensors, motion sensors, depth sensors, and/or a pointerdevice such as a computer mouse or a stylus.

In some embodiments, processing device 202 may use memory interface 204to access data and a software product stored on a memory device 226.Memory device 226 may include operating system programs for server 135that perform operating system functions when executed by the processingdevice. By way of example, the operating system programs may includeMicrosoft Windows™, Unix™ Linux™, Apple™ operating systems, personaldigital assistant (PDA) type operating systems such as Apple iOS, GoogleAndroid, Blackberry OS, or other types of operating systems.

Memory device 226 may also store communication instructions 228 tofacilitate communicating with one or more additional devices (e.g.,capturing device 125), one or more computers (e.g., output devices145A-145D) and/or one or more servers. Memory device 226 may includegraphical user interface instructions 230 to facilitate graphic userinterface processing; image processing instructions 232 to facilitateimage data processing-related processes and functions; sensor processinginstructions 234 to facilitate sensor-related processing and functions;web browsing instructions 236 to facilitate web browsing-relatedprocesses and functions; and other software instructions 238 tofacilitate other processes and functions. Each of the above identifiedinstructions and applications may correspond to a set of instructionsfor performing one or more functions described above. These instructionsneed not be implemented as separate software programs, procedures, ormodules. Memory device 226 may include additional instructions or fewerinstructions. Furthermore, various functions of server 135 may beimplemented in hardware and/or in software, including in one or moresignal processing and/or application specific integrated circuits. Forexample, server 135 may execute an image processing algorithm toidentify in received images one or more products and/or obstacles, suchas shopping carts, people, and more.

In one embodiment, memory device 226 may store database 140. Database140 may include product type model data 240 (e.g., an imagerepresentation, a list of features, a model obtained by training machinelearning algorithm using training examples, an artificial neuralnetwork, and more) that may be used to identify products in receivedimages; contract-related data 242 (e.g., planograms, promotions data,etc.) that may be used to determine if the placement of products on thestore shelves and/or the promotion execution are consistent withobligations of retail store 105; catalog data 244 (e.g., retail storechain's catalog, retail store's master file, etc.) that may be used tocheck if all product types that should be offered in retail store 105are in fact in the store, if the correct price is displayed next to anidentified product, etc.; inventory data 246 that may be used todetermine if additional products should be ordered from suppliers 115;employee data 248 (e.g., attendance data, records of training provided,evaluation and other performance-related communications, productivityinformation, etc.) that may be used to assign specific store associatesto certain tasks; and calendar data 250 (e.g., holidays, national days,international events, etc.) that may be used to determine if a possiblechange in a product model is associated with a certain event. In otherembodiments of the disclosure, database 140 may store additional typesof data or fewer types of data. Furthermore, various types of data maybe stored in one or more memory devices other than memory device 226.Throughout this disclosure, the term store associate of a retail storemay refer to any person or a robot who is tasked with performing actionsin the retail store configured to support the operation of the retailstore. Some non-limiting examples of store associates may include storeemployees, subcontractors contracted to perform such actions in theretail store, employees of entities associated with the retail store(such as suppliers of the retail store, distributers of products sold inthe retail store, etc.), people engaged through crowd sourcing toperform such actions in the retail store, robots used to perform suchactions in the retail store, and so forth.

The components and arrangements shown in FIG. 2 are not intended tolimit the disclosed embodiments. As will be appreciated by a personskilled in the art having the benefit of this disclosure, numerousvariations and/or modifications may be made to the depictedconfiguration of server 135. For example, not all components may beessential for the operation of server 135 in all cases. Any componentmay be located in any appropriate part of server 135, and the componentsmay be rearranged into a variety of configurations while providing thefunctionality of the disclosed embodiments. For example, some serversmay not include some of the elements shown in I/O system 215.

FIG. 3 is a block diagram representation of an example configuration ofcapturing device 125. In one embodiment, capturing device 125 mayinclude a processing device 302, a memory interface 304, a networkinterface 306, and a peripherals interface 308 connected to image sensor310. These components may be separated or may be integrated in one ormore integrated circuits. The various components in capturing device 125may be coupled by one or more communication buses or signal lines (e.g.,bus 300). Different aspects of the functionalities of the variouscomponents in capturing device 125 may be understood from thedescription above regarding components of server 135 having similarfunctionality.

According to one embodiment, network interface 306 may be used tofacilitate communication with server 135. Network interface 306 may bean Ethernet port connected to radio frequency receivers and transmittersand/or optical receivers and transmitters. The specific design andimplementation of network interface 306 depends on the communicationsnetwork(s) over which capturing device 125 is intended to operate. Forexample, in some embodiments, capturing device 125 may include a networkinterface 306 designed to operate over a GSM network, a GPRS network, anEDGE network, a Wi-Fi or WiMax network, a Bluetooth® network, etc. Inanother example, the implementation of network interface 306 may besimilar or identical to the implementation described above for networkinterface 206.

In the example illustrated in FIG. 3 , peripherals interface 308 ofcapturing device 125 may be connected to at least one image sensor 310associated with at least one lens 312 for capturing image data in anassociated field of view. In some configurations, capturing device 125may include a plurality of image sensors associated with a plurality oflenses 312. In other configurations, image sensor 310 may be part of acamera included in capturing device 125. According to some embodiments,peripherals interface 308 may also be connected to other sensors (notshown), such as a motion sensor, a light sensor, infrared sensor, soundsensor, a proximity sensor, a temperature sensor, a biometric sensor, orother sensing devices to facilitate related functionalities. Inaddition, a positioning sensor may also be integrated with, or connectedto, capturing device 125. For example, such positioning sensor may beimplemented using one of the following technologies: Global PositioningSystem (GPS), GLObal NAvigation Satellite System (GLONASS), Galileoglobal navigation system, BeiDou navigation system, other GlobalNavigation Satellite Systems (GNSS), Indian Regional NavigationSatellite System (IRNSS), Local Positioning Systems (LPS), Real-TimeLocation Systems (RTLS), Indoor Positioning System (IPS), Wi-Fi basedpositioning systems, cellular triangulation, and so forth. For example,the positioning sensor may be built into mobile capturing device 125,such as smartphone devices. In another example, position software mayallow mobile capturing devices to use internal or external positioningsensors (e.g., connecting via a serial port or Bluetooth).

Consistent with the present disclosure, capturing device 125 may includedigital components that collect data from image sensor 310, transform itinto an image, and store the image on a memory device 314 and/ortransmit the image using network interface 306. In one embodiment,capturing device 125 may be fixedly mountable to a store shelf or toother objects in the retail store (such as walls, ceilings, floors,refrigerators, checkout stations, displays, dispensers, rods which maybe connected to other objects in the retail store, and so forth). In oneembodiment, capturing device 125 may be split into at least two housingssuch that only image sensor 310 and lens 312 may be visible on the storeshelf, and the rest of the digital components may be located in aseparate housing. An example of this type of capturing device isdescribed below with reference to FIGS. 5-7 .

Consistent with the present disclosure, capturing device 125 may usememory interface 304 to access memory device 314. Memory device 314 mayinclude high-speed, random access memory and/or non-volatile memory suchas one or more magnetic disk storage devices, one or more opticalstorage devices, and/or flash memory (e.g., NAND, NOR) to store capturedimage data. Memory device 314 may store operating system instructions316, such as DARWIN, RTXC, LINUX, iOS, UNIX, OS X, WINDOWS, or anembedded operating system such as VXWorkS. Operating system 316 mayinclude instructions for handling basic system services and forperforming hardware dependent tasks. In some implementations, operatingsystem 316 may include a kernel (e.g., UNIX kernel, LINUX kernel, etc.).In addition, memory device 314 may store capturing instructions 318 tofacilitate processes and functions related to image sensor 310;graphical user interface instructions 320 that enables a user associatedwith capturing device 125 to control the capturing device and/or toacquire images of an area-of-interest in a retail establishment; andapplication instructions 322 to facilitate a process for monitoringcompliance of product placement or other processes.

The components and arrangements shown in FIG. 3 are not intended tolimit the disclosed embodiments. As will be appreciated by a personskilled in the art having the benefit of this disclosure, numerousvariations and/or modifications may be made to the depictedconfiguration of capturing device 125. For example, not all componentsare essential for the operation of capturing device 125 in all cases.Any component may be located in any appropriate part of capturing device125, and the components may be rearranged into a variety ofconfigurations while providing the functionality of the disclosedembodiments. For example, some capturing devices may not have lenses,and other capturing devices may include an external memory deviceinstead of memory device 314.

FIGS. 4A-4C illustrate example configurations for capturing image datain retail store 105 according to disclosed embodiments. FIG. 4Aillustrates how an aisle 400 of retail store 105 may be imaged using aplurality of capturing devices 125 fixedly connected to store shelves.FIG. 4B illustrates how aisle 400 of retail store 105 may be imagedusing a handheld communication device. FIG. 4C illustrates how aisle 400of retail store 105 may be imaged by robotic devices equipped withcameras.

With reference to FIG. 4A and consistent with the present disclosure,retail store 105 may include a plurality of capturing devices 125fixedly mounted (for example, to store shelves, walls, ceilings, floors,refrigerators, checkout stations, displays, dispensers, rods which maybe connected to other objects in the retail store, and so forth) andconfigured to collect image data. As depicted, one side of an aisle 400may include a plurality of capturing devices 125 (e.g., 125A, 125B, and125C) fixedly mounted thereon and directed such that they may captureimages of an opposing side of aisle 400. The plurality of capturingdevices 125 may be connected to an associated mobile power source (e.g.,one or more batteries), to an external power supply (e.g., a powergrid), obtain electrical power from a wireless power transmissionsystem, and so forth. As depicted in FIG. 4A, the plurality of capturingdevices 125 may be placed at different heights and at least theirvertical fields of view may be adjustable. Generally, both sides ofaisle 400 may include capturing devices 125 in order to cover both sidesof aisle 400.

Differing numbers of capturing devices 125 may be used to cover shelvingunit 402. In addition, there may be an overlap region in the field ofviews of some of capturing devices 125. For example, the fields of viewof capturing devices (e.g., adjacent capturing devices) may at leastpartially overlap with one another. In another example, one capturingdevice may have a lower field of view than the field of view of a secondcapturing device, and the two capturing devices may have at leastpartially overlapping fields of view. According to one embodiment, eachcapturing device 125 may be equipped with network interface 306 forcommunicating with server 135. In one embodiment, the plurality ofcapturing devices 125 in retail store 105 may be connected to server 135via a single WLAN. Network interface 306 may transmit informationassociated with a plurality of images captured by the plurality ofcapturing devices 125 for analysis purposes. In one example, server 135may determine an existence of an occlusion event (such as, by a person,by store equipment, such as a ladder, cart, etc.) and may provide anotification to resolve the occlusion event. In another example, server135 may determine if a disparity exists between at least one contractualobligation and product placement as determined based on automaticanalysis of the plurality of images. The transmitted information mayinclude raw images, cropped images, processed image data, data aboutproducts identified in the images, and so forth. Network interface 306may also transmit information identifying the location of the pluralitycapturing devices 125 in retail store 105.

With reference to FIG. 4B and consistent with the present disclosure,server 135 may receive image data captured by users 120. In a firstembodiment, server 135 may receive image data acquired by storeassociates. In one implementation, a handheld device of a storeassociate (e.g., capturing device 125D) may display a real-time videostream captured by the image sensor of the handheld device. Thereal-time video stream may be augmented with markings identifying to thestore associate an area-of-interest that needs manual capturing ofimages. One of the situations in which manual image capture may bedesirable may occur where the area-of-interest is outside the fields ofview of a plurality of cameras fixedly connected to store shelves inaisle 400. In other situations, manual capturing of images of anarea-of-interest may be desirable when a current set of acquired imagesis out of date (e.g., obsolete in at least one respect) or of poorquality (e.g., lacking focus, obstacles, lesser resolution, lack oflight, etc.). Additional details of this embodiment are described inApplicant's International Patent Application No. PCT/IB2018/001107,which is incorporated herein by reference.

In a second embodiment, server 135 may receive image data acquired bycrowd sourcing. In one exemplary implementation, server 135 may providea request to a detected mobile device for an updated image of thearea-of-interest in aisle 400. The request may include an incentive(e.g., $2 discount) to user 120 for acquiring the image. In response tothe request, user 120 may acquire and transmit an up-to-date image ofthe area-of-interest. After receiving the image from user 120, server135 may transmit the accepted incentive or agreed upon reward to user120. The incentive may comprise a text notification and a redeemablecoupon. In some embodiments, the incentive may include a redeemablecoupon for a product associated with the area-of-interest. Server 135may generate image-related data based on aggregation of data from imagesreceived from crowd sourcing and from images received from a pluralityof cameras fixedly connected to store shelves. Additional details ofthis embodiment are described in Applicant's International PatentApplication No. PCT/IB2017/000919, which is incorporated herein byreference.

With reference to FIG. 4C and consistent with the present disclosure,server 135 may receive image data captured by robotic devices withcameras traversing in aisle 400. The present disclosure is not limitedto the type of robotic devices used to capture images of retail store105. In some embodiments, the robotic devices may include a robot on atrack (e.g., a Cartesian robot configured to move along an edge of ashelf or in parallel to a shelf, such as capturing device 125E), a drone(e.g., capturing device 125F), and/or a robot that may move on the floorof the retail store (e.g., a wheeled robot such as capturing device125G, a legged robot, a snake-like robot, etc.). The robotic devices maybe controlled by server 135 and may be operated remotely orautonomously. In one example, server 135 may instruct capturing device125E to perform periodic scans at times when no customers or otherobstructions are identified in aisle 400. Specifically, capturing device125E may be configured to move along store shelf 404 and to captureimages of products placed on store shelf 404, products placed on storeshelf 406, or products located on shelves opposite store shelf (e.g.,store shelf 408). In another example, server 135 may instruct capturingdevice 125F to perform a scan of all the area of retail store 105 beforethe opening hour. In another example, server 135 may instruct capturingdevice 125G to capture a specific area-of-interest, similar as describedabove with reference to receiving images acquired by the storeassociates. In some embodiments, robotic capturing devices (such as 125Fand 125G) may include an internal processing unit that may allow them tonavigate autonomously within retail store 105. For example, the roboticcapturing devices may use input from sensors (e.g., image sensors, depthsensors, proximity sensors, etc.), to avoid collision with objects orpeople, and to complete the scan of the desired area of retail store105.

As discussed above with reference to FIG. 4A, the image datarepresentative of products displayed on store shelves may be acquired bya plurality of stationary capturing devices 125 fixedly mounted in theretail store. One advantage of having stationary image capturing devicesspread throughout retail store 105 is the potential for acquiringproduct images from set locations and on an ongoing basis such thatup-to-date product status may be determined for products throughout aretail store at any desired periodicity (e.g., in contrast to a movingcamera system that may acquire product images more infrequently).However, there may be certain challenges in this approach. The distancesand angles of the image capturing devices relative to the capturedproducts should be selected such as to enable adequate productidentification, especially when considered in view of image sensorresolution and/or optics specifications. For example, a capturing deviceplaced on the ceiling of retail store 105 may have sufficientresolutions and optics to enable identification of large products (e.g.,a pack of toilet paper), but may be insufficient for identifying smallerproducts (e.g., deodorant packages). The image capturing devices shouldnot occupy shelf space that is reserved for products for sale. The imagecapturing devices should not be positioned in places where there is alikelihood that their fields of view will be regularly blocked bydifferent objects. The image capturing devices should be able tofunction for long periods of time with minimum maintenance. For example,a requirement for frequent replacement of batteries may render certainimage acquisition systems cumbersome to use, especially where many imageacquisition devices are in use throughout multiple locations in a retailstore and across multiple retail stores. The image capturing devicesshould also include processing capabilities and transmissioncapabilities for providing real time or near real time image data aboutproducts. The disclosed image acquisition systems address thesechallenges.

FIG. 5A illustrates an example of a system 500 for acquiring images ofproducts in retail store 105. Throughout the disclosure, capturingdevice 125 may refer to a system, such as system 500 shown in FIG. 5A.As shown, system 500 may include a first housing 502 configured forlocation on a retail shelving unit (e.g., as illustrated in FIG. 5B),and a second housing 504 configured for location on the retail shelvingunit separate from first housing 502. The first and the second housingmay be configured for mounting on the retail shelving unit in anysuitable way (e.g., screws, bolts, clamps, adhesives, magnets,mechanical means, chemical means, etc.). In some embodiments, firsthousing 502 may include an image capture device 506 (e.g., a cameramodule that may include image sensor 310) and second housing 504 mayinclude at least one processor (e.g., processing device 302) configuredto control image capture device 506 and also to control a networkinterface (e.g., network interface 306) for communicating with a remoteserver (e.g., server 135).

System 500 may also include a data conduit 508 extending between firsthousing 502 and second housing 504. Data conduit 508 may be configuredto enable transfer of control signals from the at least one processor toimage capture device 506 and to enable collection of image data acquiredby image capture device 506 for transmission by the network interface.Consistent with the present disclosure, the term “data conduit” mayrefer to a communications channel that may include either a physicaltransmission medium such as a wire or a logical connection over amultiplexed medium such as a radio channel In some embodiments, dataconduit 508 may be used for conveying image data from image capturedevice 506 to at least one processor located in second housing 504.Consistent with one implementation of system 500, data conduit 508 mayinclude flexible printed circuits and may have a length of at leastabout 5 cm, at least about 10 cm, at least about 15 cm, etc. The lengthof data conduit 508 may be adjustable to enable placement of firsthousing 502 separately from second housing 504. For example, in someembodiments, data conduit may be retractable within second housing 504such that the length of data conduit exposed between first housing 502and second housing 504 may be selectively adjusted.

In one embodiment, the length of data conduit 508 may enable firsthousing 502 to be mounted on a first side of a horizontal store shelffacing the aisle (e.g., store shelf 510 illustrated in FIG. 5B) andsecond housing 504 to be mounted on a second side of store shelf 510that faces the direction of the ground (e.g., an underside of a storeshelf). In this embodiment, data conduit 508 may be configured to bendaround an edge of store shelf 510 or otherwise adhere/follow contours ofthe shelving unit. For example, a first portion of data conduit 508 maybe configured for location on the first side of store shelf 510 (e.g., aside facing an opposing retail shelving unit across an aisle) and asecond portion of data conduit 508 may be configured for location on asecond side of store shelf 510 (e.g., an underside of the shelf, whichin some cases may be orthogonal to the first side). The second portionof data conduit 508 may be longer than the first portion of data conduit508. Consistent with another embodiment, data conduit 508 may beconfigured for location within an envelope of a store shelf. Forexample, the envelope may include the outer boundaries of a channellocated within a store shelf, a region on an underside of an L-shapedstore shelf, a region between two store shelves, etc. Consistent withanother implementation of system 500 discussed below, data conduit 508may include a virtual conduit associated with a wireless communicationslink between first housing 502 and second housing 504.

FIG. 5B illustrates an exemplary configuration for mounting firsthousing 502 on store shelf 510. Consistent with the present disclosure,first housing 502 may be placed on store shelf 510, next to or embeddedin a plastic cover that may be used for displaying prices.Alternatively, first housing 502 may be placed or mounted on any otherlocation in retail store 105. For example, first housing 502 may beplaced or mounted on the walls, on the ceiling, on refrigerator units,on display units, and more. The location and/or orientation of firsthousing 502 may be selected such that a field of view of image capturedevice 506 may cover at least a portion of an opposing retail shelvingunit. Consistent with the present disclosure, image capture device 506may have a view angle of between 50 and 80 degrees, about 62 degrees,about 67 degrees, or about 75 degrees. Consistent with the presentdisclosure, image capture device 506 may include an image sensor havingsufficient image resolution to enable detection of text associated withlabels on an opposing retail shelving unit. In one embodiment, the imagesensor may include m*n pixels. For example, image capture device 506 mayhave an 8 MP image sensor that includes an array of 3280*2464 pixels.Each pixel may include at least one photo-voltaic cell that converts thephotons of the incident light to an electric signal. The electricalsignal may be converted to digital data by an A/D converter andprocessed by the image processor (ISP). In one embodiment, the imagesensor of image capture device 506 may be associated with a pixel sizeof between 1.1×1.1 um² and 1.7×1.7 um², for example, 1.4×1.4 um².

Consistent with the present disclosure, image capture device 506 may beassociated with a lens (e.g., lens 312) having a fixed focal lengthselected according to a distance expected to be encountered betweenretail shelving units on opposite sides of an aisle (e.g., distance d1shown in FIG. 6A) and/or according to a distance expected to beencountered between a side of a shelving unit facing the aisle on oneside of an aisle and a side of a shelving unit facing away of the aisleon the other side of the aisle (e.g., distance d2 shown in FIG. 6A). Thefocal length may also be based on any other expected distance betweenthe image acquisition device and products to be imaged. As used herein,the term “focal length” refers to the distance from the optical centerof the lens to a point where objects located at the point aresubstantially brought into focus. In contrast to zoom lenses, in fixedlenses the focus is not adjustable. The focus is typically set at thetime of lens design and remains fixed. In one embodiment, the focallength of lens 312 may be selected based on the distance between twosides of aisles in the retail store (e.g., distance d1, distance d2, andso forth). In some embodiments, image capture device 506 may include alens with a fixed focal length having a fixed value between 2.5 mm and4.5 mm, such as about 3.1 mm, about 3.4 mm, about 3.7 mm. For example,when distance d1 between two opposing retail shelving units is about 2meters, the focal length of the lens may be about 3.6 mm. Unlessindicated otherwise, the term “about” with regards to a numeric value isdefined as a variance of up to 5% with respect to the stated value. Ofcourse, image capture devices having non-fixed focal lengths may also beused depending on the requirements of certain imaging environments, thepower and space resources available, etc.

FIG. 5C illustrates an exploded view of second housing 504. In someembodiments, the network interface located in second housing 504 (e.g.,network interface 306) may be configured to transmit to server 135information associated with a plurality of images captured by imagecapture device 506. For example, the transmitted information may be usedto determine if a disparity exists between at least one contractualobligation (e.g., planogram) and product placement. In one example, thenetwork interface may support transmission speeds of 0.5 Mb/s, 1 Mb/s, 5Mb/s, or more. Consistent with the present disclosure, the networkinterface may allow different modes of operations to be selected, suchas: high-speed, slope-control, or standby. In high-speed mode,associated output drivers may have fast output rise and fall times tosupport high-speed bus rates; in slope-control, the electromagneticinterference may be reduced and the slope (i.e., the change of voltageper unit of time) may be proportional to the current output; and instandby mode, the transmitter may be switched off and the receiver mayoperate at a lower current.

Consistent with the present disclosure, second housing 504 may include apower port 512 for conveying energy from a power source to first housing502. In one embodiment, second housing 504 may include a section for atleast one mobile power source 514 (e.g., in the depicted configurationthe section is configured to house four batteries). The at least onemobile power source may provide sufficient power to enable image capturedevice 506 to acquire more than 1,000 pictures, more than 5,000pictures, more than 10,000 pictures, or more than 15,000 pictures, andto transmit them to server 135. In one embodiment, mobile power source514 located in a single second housing 504 may power two or more imagecapture devices 506 mounted on the store shelf. For example, as depictedin FIGS. 6A and 6B, a single second housing 504 may be connected to aplurality of first housings 502 with a plurality of image capturedevices 506 covering different (overlapping or non-overlapping) fieldsof view. Accordingly, the two or more image capture devices 506 may bepowered by a single mobile power source 514 and/or the data captured bytwo or more image capture devices 506 may be processed to generate apanoramic image by a single processing device located in second housing504. In addition to mobile power source 514 or as an alternative tomobile power source 514, second housing 504 may also be connected to anexternal power source. For example, second housing 504 may be mounted toa store shelf and connected to an electric power grid. In this example,power port 512 may be connected to the store shelf through a wire forproviding electrical power to image capture device 506. In anotherexample, a retail shelving unit or retail store 105 may include awireless power transmission system, and power port 512 may be connectedto a device configured to obtain electrical power from the wirelesspower transmission system. In addition, as discussed below, system 500may use power management policies to reduce the power consumption. Forexample, system 500 may use selective image capturing and/or selectivetransmission of images to reduce the power consumption or conservepower.

FIG. 6A illustrates a schematic diagram of a top view of aisle 600 inretail store 105 with multiple image acquisition systems 500 (e.g.,500A, 500B, 500C, 500D, and 500E) deployed thereon for acquiring imagesof products. Aisle 600 may include a first retail shelving unit 602 anda second retail shelving unit 604 that opposes first retail shelvingunit 602. In some embodiments, different numbers of systems 500 may bemounted on opposing retail shelving units. For example, system 500A(including first housing 502A, second housing 504A, and data conduit508A), system 500B (including first housing 502B second housing 504B,and data conduit 508B), and system 500C (including first housing 502C,second housing 504C, and data conduit 508C) may be mounted on firstretail shelving unit 602; and system 500D (including first housing502D1, first housing 502D2, second housing 504D, and data conduits 508D1and 508D2) and system 500E (including first housing 502E1, first housing502E2, second housing 504E, and data conduits 508E1 and 508E2) may bemounted on second retail shelving unit 604. Consistent with the presentdisclosure, image capture device 506 may be configured relative to firsthousing 502 such that an optical axis of image capture device 506 isdirected toward an opposing retail shelving unit when first housing 502is fixedly mounted on a retail shelving unit. For example, optical axis606 of the image capture device associated with first housing 502B maybe directed towards second retail shelving unit 604 when first housing502B is fixedly mounted on first retail shelving unit 602. A singleretail shelving unit may hold a number of systems 500 that include aplurality of image capturing devices. Each of the image capturingdevices may be associated with a different field of view directed towardthe opposing retail shelving unit. Different vantage points ofdifferently located image capture devices may enable image acquisitionrelative to different sections of a retail shelf. For example, at leastsome of the plurality of image capturing devices may be fixedly mountedon shelves at different heights. Examples of such a deployment areillustrated in FIGS. 4A and 6B.

As shown in FIG. 6A each first housing 502 may be associated with a dataconduit 508 that enables exchanging of information (e.g., image data,control signals, etc.) between the at least one processor located insecond housing 504 and image capture device 506 located in first housing502. In some embodiments, data conduit 508 may include a wiredconnection that supports data-transfer and may be used to power imagecapture device 506 (e.g., data conduit 508A, data conduit 508B, dataconduit 508D1, data conduit 508D2, data conduit 508E1, and data conduit508E2). Consistent with these embodiments, data conduit 508 may complywith a wired standard such as USB, Micro-USB, HDMI, Micro-HDMI,Firewire, Apple, etc. In other embodiments, data conduit 508 may be awireless connection, such as a dedicated communications channel betweenthe at least one processor located in second housing 504 and imagecapture device 506 located in first housing 502 (e.g., data conduit508C). In one example, the communications channel may be established bytwo Near Field Communication (NFC) transceivers. In other examples,first housing 502 and second housing 504 may include interface circuitsthat comply with other short-range wireless standards such as Bluetooth,Wi-Fi, ZigBee, etc.

In some embodiments of the disclosure, the at least one processor ofsystem 500 may cause at least one image capture device 506 toperiodically capture images of products located on an opposing retailshelving unit (e.g., images of products located on a shelf across anaisle from the shelf on which first housing 502 is mounted). The term“periodically capturing images” includes capturing an image or images atpredetermined time intervals (e.g., every minute, every 30 minutes,every 150 minutes, every 300 minutes, etc.), capturing video, capturingan image every time a status request is received, and/or capturing animage subsequent to receiving input from an additional sensor, forexample, an associated proximity sensor. Images may also be capturedbased on various other triggers or in response to various other detectedevents. In some embodiments, system 500 may receive an output signalfrom at least one sensor located on an opposing retail shelving unit.For example, system 500B may receive output signals from a sensingsystem located on second retail shelving unit 604. The output signalsmay be indicative of a sensed lifting of a product from second retailshelving unit 604 or a sensed positioning of a product on second retailshelving unit 604. In response to receiving the output signal from theat least one sensor located on second retail shelving unit 604, system500B may cause image capture device 506 to capture one or more images ofsecond retail shelving unit 604. Additional details on a sensing system,including the at least one sensor that generates output signalsindicative of a sensed lifting of a product from an opposing retailshelving unit, is discussed below with reference to FIGS. 8-10 .

Consistent with embodiments of the disclosure, system 500 may detect anobject 608 in a selected area between first retail shelving unit 602 andsecond retail shelving unit 604. Such detection may be based on theoutput of one or more dedicated sensors (e.g., motion detectors, etc.)and/or may be based on image analysis of one or more images acquired byan image acquisition device. Such images, for example, may include arepresentation of a person or other object recognizable through variousimage analysis techniques (e.g., trained neural networks, Fouriertransform analysis, edge detection, filters, face recognition, etc.).The selected area may be associated with distance d1 between firstretail shelving unit 602 and second retail shelving unit 604. Theselected area may be within the field of view of image capture device506 or an area where the object causes an occlusion of a region ofinterest (such as a shelf, a portion of a shelf being monitored, andmore). Upon detecting object 608, system 500 may cause image capturedevice 506 to forgo image acquisition while object 608 is within theselected area. In one example, object 608 may be an individual, such asa customer or a store associate. In another example, detected object 608may be an inanimate object, such as a cart, box, carton, one or moreproducts, cleaning robots, etc. In the example illustrated in FIG. 6A,system 500A may detect that object 608 has entered into its associatedfield of view (e.g., using a proximity sensor) and may instruct imagecapturing device 506 to forgo image acquisition. In alternativeembodiments, system 500 may analyze a plurality of images acquired byimage capture device 506 and identify at least one image of theplurality of images that includes a representation of object 608.Thereafter, system 500 may avoid transmission of at least part of the atleast one identified image and/or information based on the at least oneidentified image to server 135.

As shown in FIG. 6A, the at least one processor contained in a secondhousing 504 may control a plurality of image capture devices 506contained in a plurality of first housings 502 (e.g., systems 500D and500E). Controlling image capturing device 506 may include instructingimage capturing device 506 to capture an image and/or transmit capturedimages to a remote server (e.g., server 135). In some cases, each of theplurality of image capture devices 506 may have a field of view that atleast partially overlaps with a field of view of at least one otherimage capture device 506 from among plurality of image capture devices506. In one embodiment, the plurality of image capture devices 506 maybe configured for location on one or more horizontal shelves and may bedirected to substantially different areas of the opposing first retailshelving unit. In this embodiment, the at least one processor maycontrol the plurality of image capture devices such that each of theplurality of image capture devices may capture an image at a differenttime. For example, system 500E may have a second housing 504E with atleast one processor that may instruct a first image capturing devicecontained in first housing 502E1 to capture an image at a first time andmay instruct a second image capturing device contained in first housing502E2 to capture an image at a second time which differs from the firsttime. Capturing images in different times (or forwarding them to the atleast one processor at different times) may assist in processing theimages and writing the images in the memory associated with the at leastone processor.

FIG. 6B illustrates a perspective view assembly diagram depicting aportion of a retail shelving unit 620 with multiple systems 500 (e.g.,500F, 500G, 500H, 500I, and 500J) deployed thereon for acquiring imagesof products. Retail shelving unit 620 may include horizontal shelves atdifferent heights. For example, horizontal shelves 622A, 622B, and 622Care located below horizontal shelves 622D, 622E, and 622F. In someembodiments, a different number of systems 500 may be mounted on shelvesat different heights. For example, system 500F (including first housing502F and second housing 504F), system 500G (including first housing 502Gand second housing 504G), and system 500H (including first housing 502Hand second housing 504H) may be mounted on horizontal shelves associatedwith a first height; and system 500I (including first housing 502I,second housing 504I, and a projector 632) and system 500J (includingfirst housing 502J1, first housing 502J2, and second housing 504J) maybe mounted on horizontal shelves associated with a second height. Insome embodiments, retail shelving unit 620 may include a horizontalshelf with at least one designated place (not shown) for mounting ahousing of image capturing device 506. The at least one designated placemay be associated with connectors such that first housing 502 may befixedly mounted on a side of horizontal shelf 622 facing an opposingretail shelving unit using the connectors.

Consistent with the present disclosure, system 500 may be mounted on aretail shelving unit that includes at least two adjacent horizontalshelves (e.g., shelves 622A and 622B) forming a substantially continuoussurface for product placement. The store shelves may include standardstore shelves or customized store shelves. A length of each store shelf622 may be at least 50 cm, less than 200 cm, or between 75 cm to 175 cm.In one embodiment, first housing 502 may be fixedly mounted on theretail shelving unit in a slit between two adjacent horizontal shelves.For example, first housing 502G may be fixedly mounted on retailshelving unit 620 in a slit between horizontal shelf 622B and horizontalshelf 622C. In another embodiment, first housing 502 may be fixedlymounted on a first shelf and second housing 504 may be fixedly mountedon a second shelf. For example, first housing 502I may be mounted onhorizontal shelf 622D and second housing 504I may be mounted onhorizontal shelf 622E. In another embodiment, first housing 502 may befixedly mounted on a retail shelving unit on a first side of ahorizontal shelf facing the opposing retail shelving unit and secondhousing 504 may be fixedly mounted on retail shelving unit 620 on asecond side of the horizontal shelf orthogonal to the first side. Forexample, first housing 502H may mounted on a first side 624 ofhorizontal shelf 622C next to a label and second housing 504H may bemounted on a second side 626 of horizontal shelf 622C that faces down(e.g., towards the ground or towards a lower shelf). In anotherembodiment, second housing 504 may be mounted closer to the back of thehorizontal shelf than to the front of the horizontal shelf. For example,second housing 504H may be fixedly mounted on horizontal shelf 622C onsecond side 626 closer to third side 628 of the horizontal shelf 622Cthan to first side 624. Third side 628 may be parallel to first side624. As mentioned above, data conduit 508 (e.g., data conduit 508H) mayhave an adjustable or selectable length for extending between firsthousing 502 and second housing 504. In one embodiment, when firsthousing 502H is fixedly mounted on first side 624, the length of dataconduit 508H may enable second housing 604H to be fixedly mounted onsecond side 626 closer to third side 628 than to first side 624.

As mentioned above, at least one processor contained in a single secondhousing 504 may control a plurality of image capture devices 506contained in a plurality of first housings 502 (e.g., system 500J). Insome embodiments, the plurality of image capture devices 506 may beconfigured for location on a single horizontal shelf and may be directedto substantially the same area of the opposing first retail shelvingunit (e.g., system 500D in FIG. 6A). In these embodiments, the imagedata acquired by the first image capture device and the second imagecapture device may enable a calculation of depth information (e.g.,based on image parallax information) associated with at least oneproduct positioned on an opposing retail shelving unit. For example,system 500J may have single second housing 504J with at least oneprocessor that may control a first image capturing device contained infirst housing 502J1 and a second image capturing device contained infirst housing 502J2. The distance d3 between the first image capturedevice contained in first housing 502J1 and the second image capturedevice contained in first housing 502J2 may be selected based on thedistance between retail shelving unit 620 and the opposing retailshelving unit (e.g., similar to d1 and/or d2). For example, distance d3may be at least 5 cm, at least 10 cm, at least 15 cm, less than 40 cm,less than 30 cm, between about 5 cm to about 20 cm, or between about 10cm to about 15 cm. In another example, d3 may be a function of d1 and/ord2, a linear function of d1 and/or d2, a function of d1*log(d1) and/ord2*log(d2) such as a1*d1*log(d1) for some constant a1, and so forth. Thedata from the first image capturing device contained in first housing502J1 and the second image capturing device contained in first housing502J2 may be used to estimate the number of products on a store shelf ofretail shelving unit 602. In related embodiments, system 500 may controla projector (e.g., projector 632) and image capture device 506 that areconfigured for location on a single store shelf or on two separate storeshelves. For example, projector 632 may be mounted on horizontal shelf622E and image capture device 506I may be mounted on horizontal shelf622D. The image data acquired by image capture device 506 (e.g.,included in first housing 502I) may include reflections of lightpatterns projected from projector 632 on the at least one product and/orthe opposing retail shelving unit and may enable a calculation of depthinformation associated with at least one product positioned on theopposing retail shelving unit. The distance between projector 632 andthe image capture device contained in first housing 502I may be selectedbased on the distance between retail shelving unit 620 and the opposingretail shelving unit (e.g., similar to d1 and/or d2). For example, thedistance between the projector and the image capture device may be atleast 5 cm, at least 10 cm, at least 15 cm, less than 40 cm, less than30 cm, between about 5 cm to about 20 cm, or between about 10 cm toabout 15 cm. In another example, the distance between the projector andthe image capture device may be a function of d1 and/or d2, a linearfunction of d1 and/or d2, a function of d1*log(d1) and/or d2*log(d2)such as a1*d1*log(d1) for some constant a1, and so forth.

Consistent with the present disclosure, a central communication device630 may be located in retail store 105 and may be configured tocommunicate with server 135 (e.g., via an Internet connection). Thecentral communication device may also communicate with a plurality ofsystems 500 (for example, less than ten, ten, eleven, twelve, more thantwelve, and so forth). In some cases, at least one system of theplurality of systems 500 may be located in proximity to centralcommunication device 630. In the illustrated example, system 500F may belocated in proximity to central communication device 630. In someembodiments, at least some of systems 500 may communicate directly withat least one other system 500. The communications between some of theplurality of systems 500 may happen via a wired connection, such as thecommunications between system 500J and system 500I and thecommunications between system 500H and system 500G. Additionally oralternatively, the communications between some of the plurality ofsystems 500 may occur via a wireless connection, such as thecommunications between system 500G and system 500F and thecommunications between system 500I and system 500F. In some examples, atleast one system 500 may be configured to transmit captured image data(or information derived from the captured image data) to centralcommunication device 630 via at least two mediating systems 500, atleast three mediating systems 500, at least four mediating systems 500,or more. For example, system 500J may convey captured image data tocentral communication device 630 via system 500I and system 500F.

Consistent with the present disclosure, two (or more) systems 500 mayshare information to improve image acquisition. For example, system 500Jmay be configured to receive from a neighboring system 500I informationassociated with an event that system 500I had identified, and controlimage capture device 506 based on the received information. For example,system 500J may forgo image acquisition based on an indication fromsystem 500I that an object has entered or is about to enter its field ofview. Systems 500I and 500J may have overlapping fields of view ornon-overlapping fields of view. In addition, system 500J may alsoreceive (from system 500I) information that originates from centralcommunication device 630 and control image capture device 506 based onthe received information. For example, system 500I may receiveinstructions from central communication device 630 to capture an imagewhen suppler 115 inquiries about a specific product that is placed in aretail unit opposing system 500I. In some embodiments, a plurality ofsystems 500 may communicate with central communication device 630. Inorder to reduce or avoid network congestion, each system 500 mayidentify an available transmission time slot. Thereafter, each system500 may determine a default time slot for future transmissions based onthe identified transmission time slot.

FIG. 6C provides a diagrammatic representation of a retail shelving unit640 being captured by multiple systems 500 (e.g., system 500K and system500L) deployed on an opposing retail shelving unit (not shown). FIG. 6Cillustrates embodiments associated with the process of installingsystems 500 in retail store 105. To facilitate the installation ofsystem 500, each first housing 502 (e.g., first housing 502K) mayinclude an adjustment mechanism 642 for setting a field of view 644 ofimage capture device 506K such that the field of view 644 will at leastpartially encompass products placed both on a bottom shelf of retailshelving unit 640 and on a top shelf of retail shelving unit 640. Forexample, adjustment mechanism 642 may enable setting the position ofimage capture device 506K relative to first housing 502K. Adjustmentmechanism 642 may have at least two degrees of freedom to separatelyadjust manually (or automatically) the vertical field of view and thehorizontal field of view of image capture device 506K. In oneembodiment, the angle of image capture device 506K may be measured usingposition sensors associated with adjustment mechanism 642, and themeasured orientation may be used to determine if image capture device506K is positioned in the right direction. In one example, the output ofthe position sensors may be displayed on a handheld device of a person(such as a store associate) installing image capturing device 506K. Suchan arrangement may provide the store associate/installer with real timevisual feedback representative of the field of view of an imageacquisition device being installed.

In addition to adjustment mechanism 642, first housing 502 may include afirst physical adapter (not shown) configured to operate with multipletypes of image capture device 506 and a second physical adapter (notshown) configured to operate with multiple types of lenses. Duringinstallation, the first physical adapter may be used to connect asuitable image capture device 506 to system 500 according to the levelof recognition requested (e.g., detecting a barcode from products,detecting text and price from labels, detecting different categories ofproducts, etc.). Similarly, during installation, the second physicaladapter may be used to associate a suitable lens to image capture device506 according to the physical conditions at the store (e.g., thedistance between the aisles, the horizontal field of view required fromimage capture device 506, and/or the vertical field of view requiredfrom image capture device 506). The second physical adapter provides thestore associate/installer the ability to select the focal length of lens312 during installation according to the distance between retailshelving units on opposite sides of an aisle (e.g., distance d1 and/ordistance d2 shown in FIG. 6A). In some embodiments, adjustment mechanism642 may include a locking mechanism to reduce the likelihood ofunintentional changes in the field of view of image capture device 506.Additionally or alternatively, the at least one processor contained insecond housing 504 may detect changes in the field of view of imagecapture device 506 and issue a warning when a change is detected, when achange larger than a selected threshold is detected, when a change isdetected for a duration longer than a selected threshold, and so forth.

In addition to adjustment mechanism 642 and the different physicaladapters, system 500 may modify the image data acquired by image capturedevice 506 based on at least one attribute associated with opposingretail shelving unit 640. Consistent with the present disclosure, the atleast one attribute associated with retail shelving unit 640 may includea lighting condition, the dimensions of opposing retail shelving unit640, the size of products displayed on opposing retail shelving unit640, the type of labels used on opposing retail shelving unit 640, andmore. In some embodiments, the attribute may be determined, based onanalysis of one or more acquired images, by at least one processorcontained in second housing 504. Alternatively, the attribute may beautomatically sensed and conveyed to the at least one processorcontained in second housing 504. In one example, the at least oneprocessor may change the brightness of captured images based on thedetected light conditions. In another example, the at least oneprocessor may modify the image data by cropping the image such that itwill include only the products on retail shelving unit (e.g., not toinclude the floor or the ceiling), only area of the shelving unitrelevant to a selected task (such as planogram compliance check), and soforth.

Consistent with the present disclosure, during installation, system 500may enable real-time display 646 of field of view 644 on a handhelddevice 648 of a user 650 installing image capturing device 506K. In oneembodiment, real-time display 646 of field of view 644 may includeaugmented markings 652 indicating a location of a field of view 654 ofan adjacent image capture device 506L. In another embodiment, real-timedisplay 646 of field of view 644 may include augmented markings 656indicating a region of interest in opposing retail shelving unit 640.The region of interest may be determined based on a planogram,identified product type, and/or part of retail shelving unit 640. Forexample, the region of interest may include products with a greaterlikelihood of planogram incompliance. In addition, system 500K mayanalyze acquired images to determine if field of view 644 includes thearea that image capturing device 506K is supposed to monitor (forexample, from labels on opposing retail shelving unit 640, products onopposing retail shelving unit 640, images captured from other imagecapturing devices that may capture other parts of opposing retailshelving unit 640 or capture the same part of opposing retail shelvingunit 640 but in a lower resolution or at a lower frequency, and soforth). In additional embodiments, system 500 may further comprise anindoor location sensor which may help determine if the system 500 ispositioned at the right location in retail store 105.

In some embodiments, an anti-theft device may be located in at least oneof first housing 502 and second housing 504. For example, the anti-theftdevice may include a specific RF label or a pin-tag radio-frequencyidentification device, which may be the same or similar to a type ofanti-theft device that is used by retail store 105 in which system 500is located. The RF label or the pin-tag may be incorporated within thebody of first housing 502 and second housing 504 and may not be visible.In another example, the anti-theft device may include a motion sensorwhose output may be used to trigger an alarm in the case of motion ordisturbance, in case of motion that is above a selected threshold, andso forth.

FIG. 7A includes a flowchart representing an exemplary method 700 foracquiring images of products in retail store 105 in accordance withexample embodiments of the present disclosure. For purposes ofillustration, in the following description, reference is made to certaincomponents of system 500 as deployed in the configuration depicted inFIG. 6A. It will be appreciated, however, that other implementations arepossible and that other configurations may be utilized to implement theexemplary method. It will also be readily appreciated that theillustrated method may be altered to modify the order of steps, deletesteps, or further include additional steps.

At step 702, the method includes fixedly mounting on first retailshelving unit 602 at least one first housing 502 containing at least oneimage capture device 506 such that an optical axis (e.g., optical axis606) of at least one image capture device 506 is directed to secondretail shelving unit 604. In one embodiment, fixedly mounting firsthousing 502 on first retail shelving unit 602 may include placing firsthousing 502 on a side of store shelf 622 facing second retail shelvingunit 604. In another embodiment, fixedly mounting first housing 502 onretail shelving unit 602 may include placing first housing 502 in a slitbetween two adjacent horizontal shelves. In some embodiments, the methodmay further include fixedly mounting on first retail shelving unit 602at least one projector (such as projector 632) such that light patternsprojected by the at least one projector are directed to second retailshelving unit 604. In one embodiment, the method may include mountingthe at least one projector to first retail shelving unit 602 at aselected distance to first housing 502 with image capture device 506. Inone embodiment, the selected distance may be at least 5 cm, at least 10cm, at least 15 cm, less than 40 cm, less than 30 cm, between about 5 cmto about 20 cm, or between about 10 cm to about 15 cm. In oneembodiment, the selected distance may be calculated according to adistance between to first retail shelving unit 602 and second retailshelving unit 604, such as d1 and/or d2, for example selecting thedistance to be a function of d1 and/or d2, a linear function of d1and/or d2, a function of d1*log(d1) and/or d2*log(d2) such asa1*d1*log(d1) for some constant a1, and so forth.

At step 704, the method includes fixedly mounting on first retailshelving unit 602 second housing 504 at a location spaced apart from theat least one first housing 502, second housing 504 may include at leastone processor (e.g., processing device 302). In one embodiment, fixedlymounting second housing 504 on the retail shelving unit may includeplacing second housing 504 on a different side of store shelf 622 thanthe side first housing 502 is mounted on.

At step 706, the method includes extending at least one data conduit 508between at least one first housing 502 and second housing 504. In oneembodiment, extending at least one data conduit 508 between at least onefirst housing 502 and second housing 504 may include adjusting thelength of data conduit 508 to enable first housing 502 to be mountedseparately from second housing 504. At step 708, the method includescapturing images of second retail shelving unit 604 using at least oneimage capture device 506 contained in at least one first housing 502(e.g., first housing 502A, first housing 502B, or first housing 502C).In one embodiment, the method further includes periodically capturingimages of products located on second retail shelving unit 604. Inanother embodiment the method includes capturing images of second retailshelving unit 604 after receiving a trigger from at least one additionalsensor in communication with system 500 (wireless or wired).

At step 710, the method includes transmitting at least some of thecaptured images from second housing 504 to a remote server (e.g., server135) configured to determine planogram compliance relative to secondretail shelving unit 604. In some embodiments, determining planogramcompliance relative to second retail shelving unit 604 may includedetermining at least one characteristic of planogram compliance based ondetected differences between the at least one planogram and the actualplacement of the plurality of product types on second retail shelvingunit 604. Consistent with the present disclosure, the characteristic ofplanogram compliance may include at least one of: product facing,product placement, planogram compatibility, price correlation, promotionexecution, product homogeneity, restocking rate, and planogramcompliance of adjacent products.

FIG. 7B provides a flowchart representing an exemplary method 720 foracquiring images of products in retail store 105, in accordance withexample embodiments of the present disclosure. For purposes ofillustration, in the following description, reference is made to certaincomponents of system 500 as deployed in the configuration depicted inFIG. 6A. It will be appreciated, however, that other implementations arepossible and that other configurations may be utilized to implement theexemplary method. It will also be readily appreciated that theillustrated method may be altered to modify the order of steps, deletesteps, or further include additional steps.

At step 722, at least one processor contained in a second housing mayreceive from at least one image capture device contained in at least onefirst housing fixedly mounted on a retail shelving unit a plurality ofimages of an opposing retail shelving unit. For example, at least oneprocessor contained in second housing 504A may receive from at least oneimage capture device 506 contained in first housing 502A (fixedlymounted on first retail shelving unit 602) a plurality of images ofsecond retail shelving unit 604. The plurality of images may be capturedand collected during a period of time (e.g., a minute, an hour, sixhours, a day, a week, or more).

At step 724, the at least one processor contained in the second housingmay analyze the plurality of images acquired by the at least one imagecapture device. In one embodiment, at least one processor contained insecond housing 504A may use any suitable image analysis technique (forexample, object recognition, object detection, image segmentation,feature extraction, optical character recognition (OCR), object-basedimage analysis, shape region techniques, edge detection techniques,pixel-based detection, artificial neural networks, convolutional neuralnetworks, etc.) to identify objects in the plurality of images. In oneexample, the at least one processor contained in second housing 504A maydetermine the number of products located in second retail shelving unit604. In another example, the at least one processor contained in secondhousing 504A may detect one or more objects in an area between firstretail shelving unit 602 and second retail shelving unit 604.

At step 726, the at least one processor contained in the second housingmay identify in the plurality of images a first image that includes arepresentation of at least a portion of an object located in an areabetween the retail shelving unit and the opposing retail shelving unit.In step 728, the at least one processor contained in the second housingmay identify in the plurality of images a second image that does notinclude any object located in an area between the retail shelving unitand the opposing retail shelving unit. In one example, the object in thefirst image may be an individual, such as a customer or a storeassociate. In another example, the object in the first image may be aninanimate object, such as carts, boxes, products, etc.

At step 730, the at least one processor contained in the second housingmay instruct a network interface contained in the second housing,fixedly mounted on the retail shelving unit separate from the at leastone first housing, to transmit the second image to a remote server andto avoid transmission of the first image to the remote server. Inaddition, the at least one processor may issue a notification when anobject blocks the field of view of the image capturing device for morethan a predefined period of time (e.g., at least 30 minutes, at least 75minutes, at least 150 minutes).

Embodiments of the present disclosure may automatically assesscompliance of one or more store shelves with a planogram. For example,embodiments of the present disclosure may use signals from one or moresensors to determine placement of one or more products on store shelves.The disclosed embodiments may also use one or more sensors to determineempty spaces on the store shelves. The placements and empty spaces maybe automatically assessed against a digitally encoded planogram. Aplanogram refers to any data structure or specification that defines atleast one product characteristic relative to a display structureassociated with a retail environment (such as store shelf or area of oneor more shelves). Such product characteristics may include, among otherthings, quantities of products with respect to areas of the shelves,product configurations or product shapes with respect to areas of theshelves, product arrangements with respect to areas of the shelves,product density with respect to areas of the shelves, productcombinations with respect to areas of the shelves, etc. Althoughdescribed with reference to store shelves, embodiments of the presentdisclosure may also be applied to end caps or other displays; bins,shelves, or other organizers associated with a refrigerator or freezerunits; or any other display structure associated with a retailenvironment.

The embodiments disclosed herein may use any sensors configured todetect one or more parameters associated with products (or a lackthereof). For example, embodiments may use one or more of pressuresensors, weight sensors, light sensors, resistive sensors, capacitivesensors, inductive sensors, vacuum pressure sensors, high pressuresensors, conductive pressure sensors, infrared sensors, photo-resistorsensors, photo-transistor sensors, photo-diode sensors, ultrasonicsensors, or the like. Some embodiments may use a plurality of differentkinds of sensors, for example, associated with the same or overlappingareas of the shelves and/or associated with different areas of theshelves. Some embodiments may use a plurality of sensors configured tobe placed adjacent a store shelf, configured for location on the storeshelf, configured to be attached to, or configured to be integrated withthe store shelf. In some cases, at least part of the plurality ofsensors may be configured to be placed next to a surface of a storeshelf configured to hold products. For example, the at least part of theplurality of sensors may be configured to be placed relative to a partof a store shelf such that the at least part of the plurality of sensorsmay be positioned between the part of a store shelf and products placedon the part of the shelf. In another embodiment, the at least part ofthe plurality of sensors may be configured to be placed above and/orwithin and/or under the part of the shelf.

In one example, the plurality of sensors may include light detectorsconfigured to be located such that a product placed on the part of theshelf may block at least some of the ambient light from reaching thelight detectors. The data received from the light detectors may beanalyzed to detect a product or to identify a product based on the shapeof a product placed on the part of the shelf. In one example, the systemmay identify the product placed above the light detectors based on datareceived from the light detectors that may be indicative of at leastpart of the ambient light being blocked from reaching the lightdetectors. Further, the data received from the light detectors may beanalyzed to detect vacant spaces on the store shelf. For example, thesystem may detect vacant spaces on the store shelf based on the receiveddata that may be indicative of no product being placed on a part of theshelf. In another example, the plurality of sensors may include pressuresensors configured to be located such that a product placed on the partof the shelf may apply detectable pressure on the pressure sensors.Further, the data received from the pressure sensors may be analyzed todetect a product or to identify a product based on the shape of aproduct placed on the part of the shelf. In one example, the system mayidentify the product placed above the pressure sensors based on datareceived from the pressure sensors being indicative of pressure beingapplied on the pressure sensors. In addition, the data from the pressuresensors may be analyzed to detect vacant spaces on the store shelf, forexample based on the readings being indicative of no product beingplaced on a part of the shelf, for example, when the pressure readingsare below a selected threshold. Consistent with the present disclosure,inputs from different types of sensors (such as pressure sensors, lightdetectors, etc.) may be combined and analyzed together, for example todetect products placed on a store shelf, to identify shapes of productsplaced on a store shelf, to identify types of products placed on a storeshelf, to identify vacant spaces on a store shelf, and so forth.

With reference to FIG. 8A and consistent with the present disclosure, astore shelf 800 may include a plurality of detection elements, e.g.,detection elements 801A and 801B. In the example of FIG. 8A, detectionelements 801A and 801B may comprise pressure sensors and/or other typeof sensors for measuring one or more parameters (such as resistance,capacitance, or the like) based on physical contact (or lack thereof)with products, e.g., product 803A and product 803B. Additionally oralternatively, detection elements configured to measure one or moreparameters (such as current induction, magnetic induction, visual orother electromagnetic reflectance, visual or other electromagneticemittance, or the like) may be included to detect products based onphysical proximity (or lack thereof) to products. Consistent with thepresent disclosure, the plurality of detection elements may beconfigured for location on shelf 800. The plurality of detectionelements may be configured to detect placement of products when theproducts are placed above at least part of the plurality of detectionelements. Some embodiments of the disclosure, however, may be performedwhen at least some of the detection elements may be located next toshelf 800 (e.g., for magnetometers or the like), across from shelf 800(e.g., for image sensors or other light sensors, light detection andranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, orthe like), above shelf 800 (e.g., for acoustic sensors or the like),below shelf 800 (e.g., for pressure sensors or the like), or any otherappropriate spatial arrangement. Although depicted as standalone unitsin the example of FIG. 8A, the plurality of detection elements may formpart of a fabric (e.g., a smart fabric or the like), and the fabric maybe positioned on a shelf to take measurements. For example, two or moredetection elements may be integrated together into a single structure(e.g., disposed within a common housing, integrated together within afabric or mat, etc.). In some examples, detection elements (such asdetection elements 801A and 801B) may be placed adjacent to (or placedon) store shelves as described above. Some examples of detectionelements may include pressure sensors and/or light detectors configuredto be placed above and/or within and/or under a store shelf as describedabove.

Detection elements associated with shelf 800 may be associated withdifferent areas of shelf 800. For example, detection elements 801A and801B are associated with area 805A while other detection elements areassociated with area 805B. Although depicted as rows, areas 805A and805B may comprise any areas of shelf 800, whether contiguous (e.g., asquare, a rectangular, or other regular or irregular shape) or not(e.g., a plurality of rectangles or other regular and/or irregularshapes). Such areas may also include horizontal regions between shelves(as shown in FIG. 8A) or may include vertical regions that include areasof multiple different shelves (e.g., columnar regions spanning overseveral different horizontally arranged shelves). In some examples, theareas may be part of a single plane. In some examples, each area may bepart of a different plane. In some examples, a single area may be partof a single plane or be divided across multiple planes.

One or more processors (e.g., processing device 202) configured tocommunicate with the detection elements (e.g., detection elements 801Aand 801B) may detect first signals associated with a first area (e.g.,areas 805A and/or 805B) and second signals associated with a secondarea. In some embodiments, the first area may, in part, overlap with thesecond area. For example, one or more detection elements may beassociated with the first area as well as the second area and/or one ormore detection elements of a first type may be associated with the firstarea while one or more detection elements of a second type may beassociated with the second area overlapping, at least in part, the firstarea. In other embodiments, the first area and the second area may bespatially separate from each other.

The one or more processors may, using the first and second signals,determine that one or more products have been placed in the first areawhile the second area includes at least one empty area. For example, ifthe detection elements include pressure sensors, the first signals mayinclude weight signals that match profiles of particular products (suchas the mugs or plates depicted in the example of FIG. 8A), and thesecond signals may include weight signals indicative of the absence ofproducts (e.g., by being equal to or within a threshold of a defaultvalue such as atmospheric pressure or the like). The disclosed weightsignals may be representative of actual weight values associated with aparticular product type or, alternatively, may be associated with arelative weight value sufficient to identify the product and/or toidentify the presence of a product. In some cases, the weight signal maybe suitable for verifying the presence of a product regardless ofwhether the signal is also sufficient for product identification. Inanother example, if the detection elements include light detectors (asdescribed above), the first signals may include light signals that matchprofiles of particular products (such as the mugs or plates depicted inthe example of FIG. 8A), and the second signals may include lightsignals indicative of the absence of products (e.g., by being equal toor within a threshold of a default value such as values corresponding toambient light or the like). For example, the first light signals may beindicative of ambient light being blocked by particular products, whilethe second light signals may be indicative of no product blocking theambient light. The disclosed light signals may be representative ofactual light patterns associated with a particular product type or,alternatively, may be associated with light patterns sufficient toidentify the product and/or to identify the presence of a product.

The one or more processors may similarly process signals from othertypes of sensors. For example, if the detection elements includeresistive or inductive sensors, the first signals may includeresistances, voltages, and/or currents that match profiles of particularproducts (such as the mugs or plates depicted in the example of FIG. 8Aor elements associated with the products, such as tags, etc.), and thesecond signals may include resistances, voltages, and/or currentsindicative of the absence of products (e.g., by being equal to or withina threshold of a default value such as atmospheric resistance, a defaultvoltage, a default current, corresponding to ambient light, or thelike). In another example, if the detection elements include acoustics,LIDAR, RADAR, or other reflective sensors, the first signals may includepatterns of returning waves (whether sound, visible light, infraredlight, radio, or the like) that match profiles of particular products(such as the mugs or plates depicted in the example of FIG. 8A), and thesecond signals may include patterns of returning waves (whether sound,visible light, infrared light, radio, or the like) indicative of theabsence of products (e.g., by being equal to or within a threshold of apattern associated with an empty shelf or the like).

Any of the profile matching described above may include direct matchingof a subject to a threshold. For example, direct matching may includetesting one or more measured values against the profile value(s) withina margin of error; mapping a received pattern onto a profile patternwith a residual having a maximum, minimum, integral, or the like withinthe margin of error; performing an autocorrelation, Fourier transform,convolution, or other operation on received measurements or a receivedpattern and comparing the resultant values or function against theprofile within a margin of error; or the like. Additionally oralternatively, profile matching may include fuzzy matching betweenmeasured values and/or patterns and a database of profiles such that aprofile with a highest level of confidence according to the fuzzysearch. Moreover, as depicted in the example of FIG. 8A, products, suchas product 803B, may be stacked and thus associated with a differentprofile when stacked than when standalone.

Any of the profile matching described above may include use of one ormore machine learning techniques. For example, one or more artificialneural networks, random forest models, or other models trained onmeasurements annotated with product identifiers may process themeasurements from the detection elements and identify productstherefrom. In such embodiments, the one or more models may useadditional or alternative input, such as images of the shelf (e.g., fromcapturing devices 125 of FIGS. 4A-4C explained above) or the like.

Based on detected products and/or empty spaces, determined using thefirst signals and second signals, the one or more processors maydetermine one or more aspects of planogram compliance. For example, theone or more processors may identify products and their locations on theshelves, determine quantities of products within particular areas (e.g.,identifying stacked or clustered products), identify facing directionsassociated with the products (e.g., whether a product is outward facing,inward facing, askew, or the like), or the like. Identification of theproducts may include identifying a product type (e.g., a bottle of soda,a loaf of broad, a notepad, or the like) and/or a product brand (e.g., aCoca-Cola® bottle instead of a Sprite® bottle, a Starbucks® coffeetumbler instead of a Tervis® coffee tumbler, or the like). Productfacing direction and/or orientation, for example, may be determinedbased on a detected orientation of an asymmetric shape of a product baseusing pressure sensitive pads, detected density of products, etc. Forexample, the product facing may be determined based on locations ofdetected product bases relative to certain areas of a shelf (e.g., alonga front edge of a shelf), etc. Product facing may also be determinedusing image sensors, light sensors, or any other sensor suitable fordetecting product orientation.

The one or more processors may generate one or more indicators of theone or more aspects of planogram compliance. For example, an indicatormay comprise a data packet, a data file, or any other data structureindicating any variations from a planogram, e.g., with respect toproduct placement such as encoding intended coordinates of a product andactual coordinates on the shelf, with respect to product facingdirection and/or orientation such as encoding indicators of locationsthat have products not facing a correct direction and/or in an undesiredorientation, or the like.

In addition to or as an alternative to determining planogram compliance,the one or more processors may detect a change in measurements from oneor more detection elements. Such measurement changes may trigger aresponse. For example, a change of a first type may trigger capture ofat least one image of the shelf (e.g., using capturing devices 125 ofFIGS. 4A-4C explained above) while a detected change of a second typemay cause the at least one processor to forgo such capture. A first typeof change may, for example, indicate the moving of a product from onelocation on the shelf to another location such that planogram compliancemay be implicated. In such cases, it may be desired to capture an imageof the product rearrangement in order to assess or reassess productplanogram compliance. In another example, a first type of change mayindicate the removal of a product from the shelf, e.g., by a storeassociate due to damage, by a customer to purchase, or the like. On theother hand, a second type of change may, for example, indicate theremoval and replacement of a product to the same (within a margin oferror) location on the shelf, e.g., by a customer to inspect the item.In cases where products are removed from a shelf, but then replaced onthe shelf (e.g., within a particular time window), the system may forgoa new image capture, especially if the replaced product is detected in alocation similar to or the same as its recent, original position.

With reference to FIG. 8B and consistent with the present disclosure, astore shelf 850 may include a plurality of detection elements, e.g.,detection elements 851A and 851B. In the example of FIG. 8B, detectionelements 851A and 851B may comprise light sensors and/or other sensorsmeasuring one or more parameters (such as visual or otherelectromagnetic reflectance, visual or other electromagnetic emittance,or the like) based on electromagnetic waves from products, e.g., product853A and product 853B. Additionally or alternatively, as explained abovewith respect to FIG. 8B, detection elements 851A and 851B may comprisepressure sensors, other sensors measuring one or more parameters (suchas resistance, capacitance, or the like) based on physical contact (orlack thereof) with the products, and/or other sensors that measure oneor more parameters (such as current induction, magnetic induction,visual or other electromagnetic reflectance, visual or otherelectromagnetic emittance, or the like) based on physical proximity (orlack thereof) to products.

Moreover, although depicted as located on shelf 850, some detectionelements may be located next to shelf 850 (e.g., for magnetometers orthe like), across from shelf 850 (e.g., for image sensors or other lightsensors, light detection and ranging (LIDAR) sensors, radio detectionand ranging (RADAR) sensors, or the like), above shelf 850 (e.g., foracoustic sensors or the like), below shelf 850 (e.g., for pressuresensors, light detectors, or the like), or any other appropriate spatialarrangement. Further, although depicted as standalone in the example ofFIG. 8B, the plurality of detection elements may form part of a fabric(e.g., a smart fabric or the like), and the fabric may be positioned ona shelf to take measurements.

Detection elements associated with shelf 850 may be associated withdifferent areas of shelf 850, e.g., area 855A, area 855B, or the like.Although depicted as rows, areas 855A and 855B may comprise any areas ofshelf 850, whether contiguous (e.g., a square, a rectangular, or otherregular or irregular shape) or not (e.g., a plurality of rectangles orother regular and/or irregular shapes).

One or more processors (e.g., processing device 202) in communicationwith the detection elements (e.g., detection elements 851A and 851B) maydetect first signals associated with a first area and second signalsassociated with a second area. Any of the processing of the first andsecond signals described above with respect to FIG. 8A may similarly beperformed for the configuration of FIG. 8B.

In both FIGS. 8A and 8B, the detection elements may be integral to theshelf, part of a fabric or other surface configured for positioning onthe shelf, or the like. Power and/or data cables may form part of theshelf, the fabric, the surface, or be otherwise connected to thedetection elements. Additionally or alternatively, as depicted in FIGS.8A and 8B, individual sensors may be positioned on the shelf. Forexample, the power and/or data cables may be positioned under the shelfand connected through the shelf to the detection elements. In anotherexample, power and/or data may be transmitted wirelessly to thedetection elements (e.g., to wireless network interface controllersforming part of the detection elements). In yet another example, thedetection elements may include internal power sources (such as batteriesor fuel cells).

With reference to FIG. 9 and consistent with the present disclosure, thedetection elements described above with reference to FIGS. 8A and 8B maybe arranged on rows of the shelf in any appropriate configuration. Allof the arrangements of FIG. 9 are shown as a top-down view of a row(e.g., area 805A, area 805B, area 855A, area 855B, or the like) on theshelf. For example, arrangements 910 and 940 are both uniformdistributions of detection elements within a row. However, arrangement910 is also uniform throughout the depth of the row while arrangement940 is staggered. Both arrangements may provide signals that representproducts on the shelf in accordance with spatially uniform measurementlocations. As further shown in FIG. 9 , arrangements 920, 930, 950, and960 cluster detection elements near the front (e.g., a facing portion)of the row. Arrangement 920 includes detection elements at a frontportion while arrangement 930 includes detection elements in a largerportion of the front of the shelf. Such arrangements may save power andprocessing cycles by having fewer detection elements on a back portionof the shelf. Arrangements 950 and 960 include some detection elementsin a back portion of the shelf but these elements are arranged lessdense than detection elements in the front. Such arrangements may allowfor detections in the back of the shelf (e.g., a need to restockproducts, a disruption to products in the back by a customer or a storeassociate, or the like) while still using less power and fewerprocessing cycles than arrangements 910 and 940. And, such arrangementsmay include a higher density of detection elements in regions of theshelf (e.g., a front edge of the shelf) where product turnover rates maybe higher than in other regions (e.g., at areas deeper into a shelf),and/or in regions of the shelf where planogram compliance is especiallyimportant.

FIG. 10A is a flow chart, illustrating an exemplary method 1000 formonitoring planogram compliance on a store shelf, in accordance with thepresently disclosed subject matter. It is contemplated that method 1000may be used with any of the detection element arrays discussed abovewith reference to, for example, FIGS. 8A, 8B and 9 . The order andarrangement of steps in method 1000 is provided for purposes ofillustration. As will be appreciated from this disclosure, modificationsmay be made to process 1000, for example, adding, combining, removing,and/or rearranging one or more steps of process 1000.

Method 1000 may include a step 1005 of receiving first signals from afirst subset of detection elements (e.g., detection elements 801A and801B of FIG. 8A) from among the plurality of detection elements afterone or more of a plurality of products (e.g., products 803A and 803B)are placed on at least one area of the store shelf associated with thefirst subset of detection elements. As explained above with respect toFIGS. 8A and 8B, the plurality of detection elements may be embeddedinto a fabric configured to be positioned on the store shelfAdditionally or alternatively, the plurality of detection elements maybe configured to be integrated with the store shelf. For example, anarray of pressure sensitive elements (or any other type of detector) maybe fabricated as part of the store shelf. In some examples, theplurality of detection elements may be configured to placed adjacent to(or located on) store shelves, as described above.

As described above with respect to arrangements 910 and 940 of FIG. 9 ,the plurality of detection elements may be substantially uniformlydistributed across the store shelf. Alternatively, as described abovewith respect to arrangements 920, 930, 950, and 960 of FIG. 9 , theplurality of detection elements may be distributed relative to the storeshelf such that a first area of the store shelf has a higher density ofdetection elements than a second area of the store shelf. For example,the first area may comprise a front portion of the shelf, and the secondarea may comprise a back portion of the shelf.

In some embodiments, such as those including pressure sensors or othercontact sensors as depicted in the example of FIG. 8A, step 1005 mayinclude receiving the first signals from the first subset of detectionelements as the plurality of products are placed above the first subsetof detection elements. In some embodiments where the plurality ofdetection elements includes pressure detectors, the first signals may beindicative of pressure levels detected by pressure detectorscorresponding to the first subset of detection elements after one ormore of the plurality of products are placed on the at least one area ofthe store shelf associated with the first subset of detection elements.For example, the first signals may be indicative of pressure levelsdetected by pressure detectors corresponding to the first subset ofdetection elements after stocking at least one additional product abovea product previously positioned on the shelf, removal of a product fromthe shelf, or the like. In other embodiments where the plurality ofdetection elements includes light detectors, the first signals may beindicative of light measurements made with respect to one or more of theplurality of products placed on the at least one area of the store shelfassociated with the first subset of detection elements. Specifically,the first signals may be indicative of at least part of the ambientlight being blocked from reaching the light detectors by the one or moreof the plurality of products.

In embodiments including proximity sensors as depicted in the example ofFIG. 8B, step 1005 may include receiving the first signals from thefirst subset of detection elements as the plurality of products areplaced below the first subset of detection elements. In embodimentswhere the plurality of detection elements include proximity detectors,the first signals may be indicative of proximity measurements made withrespect to one or more of the plurality of products placed on the atleast one area of the store shelf associated with the first subset ofdetection elements.

Method 1000 may include step 1010 of using the first signals to identifyat least one pattern associated with a product type of the plurality ofproducts. For example, any of the pattern matching techniques describedabove with respect to FIGS. 8A and 8B may be used for identification. Apattern associated with a product type may include a pattern (e.g., acontinuous ring, a discontinuous ring of a certain number of points, acertain shape, etc.) associated with a base of a single product. Thepattern associated with a product type may also be formed by a group ofproducts. For example, a six pack of soda cans may be associated with apattern including a 2×3 array of continuous rings associated with thesix cans of that product type. Additionally, a grouping of two literbottles may form a detectable pattern including an array (whetheruniform, irregular, or random) of discontinuous rings of pressurepoints, where the rings have a diameter associated with a particular2-liter product. Various other types of patterns may also be detected(e.g., patterns associated with different product types arrangedadjacent to one another, patterns associated with solid shapes (such asa rectangle of a boxed product), etc.). In another example, anartificial neural network configured to recognize product types may beused to analyze the signals received by step 1005 (such as signals frompressure sensors, from light detectors, from contact sensors, and soforth) to determine product types associated with products placed on anarea of a shelf (such as an area of a shelf associated with the firstsubset of detection elements). In yet another example, a machinelearning algorithm trained using training examples to recognize producttypes may be used to analyze the signals received by step 1005 (such assignals from pressure sensors, from light detectors, from contactsensors, and so forth) to determine product types associated withproducts placed on an area of a shelf (such as an area of a shelfassociated with the first subset of detection elements).

In some embodiments, step 1010 may further include accessing a memorystoring data (e.g., memory device 226 of FIG. 2 and/or memory device 314of FIG. 3A) associated with patterns of different types of products. Insuch embodiments, step 1010 may include using the first signals toidentify at least one product of a first type using a first pattern (ora first product model) and at least one product of a second type using asecond pattern (or a second product model). For example, the first typemay include one brand (such as Coca-Cola® or Folgers®) while the secondtype may include another brand (such as Pepsi® or Maxwell House®). Inthis example, a size, shape, point spacing, weight, resistance or otherproperty of the first brand may be different from that of the secondbrand such that the detection elements may differentiate the brands.Such characteristics may also be used to differentiate like-branded, butdifferent products from one another (e.g., a 12-ounce can of Coca Cola,versus a 16 ounce bottle of Coca Cola, versus a 2-liter bottle of CocaCola). For example, a soda may have a base detectable by a pressuresensitive pad as a continuous ring. Further, the can of soda may beassociated with a first weight signal having a value recognizable asassociated with such a product. A 16 ounce bottle of soda may beassociated with a base having four or five pressure points, which apressure sensitive pad may detect as arranged in a pattern associatedwith a diameter typical of such a product. The 16 ounce bottle of sodamay also be associated with a second weight signal having a value higherthan the weight signal associated with the 12 ounce can of soda. Furtherstill, a 2 liter bottle of soda may be associated with a base having aring, four or five pressure points, etc. that a pressure sensitive padmay detect as arranged in a pattern associated with a diameter typicalof such a product. The 2 liter bottle of soda may be associated with aweight signal having a value higher than the weight signal associatedwith the 12 ounce can of soda and 16 ounce bottle of soda.

In the example of FIG. 8B, the different bottoms of product 853A andproduct 853B may be used to differentiate the products from each other.For example, detection elements such as pressure sensitive pads may beused to detect a product base shape and size (e.g., ring, pattern ofpoints, asymmetric shape, base dimensions, etc.). Such a base shape andsize may be used (optionally, together with one or more weight signals)to identify a particular product. The signals may also be used toidentify and/or distinguish product types from one another. For example,a first type may include one category of product (such as soda cans)while a second type may include a different category of product (such asnotepads). In another example, detection elements such as lightdetectors may be used to detect a product based on a pattern of lightreadings indicative of a product blocking at least part of the ambientlight from reaching the light detectors. Such pattern of light readingsmay be used to identify product type and/or product category and/orproduct shape. For example, products of a first type may block a firstsubset of light frequencies of the ambient light from reaching the lightdetectors, while products of a second type may block a second subset oflight frequencies of the ambient light from reaching the light detectors(the first subset and second subset may differ). In this case the typeof the products may be determined based on the light frequenciesreaching the light detectors. In another example, products of a firsttype may have a first shape of shades and therefore may block ambientlight from reaching light detectors arranged in one shape, whileproducts of a second type may have a second shape of shades andtherefore may block ambient light from reaching light detectors arrangedin another shape. In this case the type of the products may bedetermined based on the shape of blocked ambient light. Any of thepattern matching techniques described above may be used for theidentification.

Additionally or alternatively, step 1010 may include using the at leastone pattern to determine a number of products placed on the at least onearea of the store shelf associated with the first subset of detectionelements. For example, any of the pattern matching techniques describedabove may be used to identify the presence of one or more product typesand then to determine the number of products of each product type (e.g.,by detecting a number of similarly sized and shaped product bases andoptionally by detecting weight signals associated with each detectedbase). In another example, an artificial neural network configured todetermine the number of products of selected product types may be usedto analyze the signals received by step 1005 (such as signals frompressure sensors, from light detectors, from contact sensors, and soforth) to determine the number of products of selected product typesplaced on an area of a shelf (such as an area of a shelf associated withthe first subset of detection elements). In yet another example, amachine learning algorithm trained using training examples to determinethe number of products of selected product types may be used to analyzethe signals received by step 1005 (such as signals from pressuresensors, from light detectors, from contact sensors, and so forth) todetermine the number of products of selected product types placed on anarea of a shelf (such as an area of a shelf associated with the firstsubset of detection elements). Additionally or alternatively, step 1010may include extrapolating from a stored pattern associated with a singleproduct (or type of product) to determine the number of productsmatching the first signals. In such embodiments, step 1010 may furtherinclude determining, for example based on product dimension data storedin a memory, a number of additional products that may be placed on theat least one area of the store shelf associated with the second subsetof detection elements. For example, step 1010 may include extrapolatingbased on stored dimensions of each product and stored dimensions of theshelf area to determine an area and/or volume available for additionalproducts. Step 1010 may further include extrapolation of the number ofadditional products based on the stored dimensions of each product anddetermined available area and/or volume.

Method 1000 may include step 1015 of receiving second signals from asecond subset of detection elements (e.g., detection elements 851A and851B of FIG. 8B) from among the plurality of detection elements, thesecond signals being indicative of no products being placed on at leastone area of the store shelf associated with the second subset ofdetection elements. Using this information, method 1000 may include step1020 of using the second signals to determine at least one empty spaceon the store shelf. For example, any of the pattern matching techniquesdescribed above may be used to determine that the second signals includedefault values or other values indicative of a lack of product incertain areas associated with a retail store shelf. A default value maybe include, for example, a pressure signal associated with an un-loadedpressure sensor or pressure sensitive mat, indicating that no product islocated in a certain region of a shelf In another example, a defaultvalue may include signals from light detectors corresponding to ambientlight, indicating that no product is located in a certain region of ashelf.

Method 1000 may include step 1025 of determining, based on the at leastone pattern associated with a detected product and the at least oneempty space, at least one aspect of planogram compliance. As explainedabove with respect to FIGS. 8A and 8B, the aspect of planogramcompliance may include the presence or absence of particular products(or brands), locations of products on the shelves, quantities ofproducts within particular areas (e.g., identifying stacked or clusteredproducts), facing directions associated with the products (e.g., whethera product is outward facing, inward facing, askew, or the like), or thelike. A planogram compliance determination may be made, for example, bydetermining a number of empty spaces on a shelf and determining alocation of the empty spaces on a shelf. The planogram determination mayalso include determining weight signal magnitudes associated withdetected products at the various detected non-empty locations. Thisinformation may be used by the one or more processors in determiningwhether a product facing specification has been satisfied (e.g., whethera front edge of a shelf has a suitable number of products or suitabledensity of products), whether a specified stacking density has beenachieved (e.g., by determining a pattern of detected products and weightsignals of the detected products to determine how many products arestacked at each location), whether a product density specification hasbeen achieved (e.g., by determining a ratio of empty locations toproduct-present locations), whether products of a selected product typeare located in a selected area of the shelf, whether all productslocated in a selected area of the shelf are of a selected product type,whether a selected number of products (or a selected number of productsof a selected product type) are located in a selected area of the shelf,whether products located in a selected area of a shelf are positioned ina selected orientation, or whether any other aspect of one or moreplanograms has been achieved.

For example, the at least one aspect may include product homogeneity,and step 1025 may further include counting occurrences where a productof the second type is placed on an area of the store shelf associatedwith the first type of product. For example, by accessing a memoryincluding base patterns (or any other type of pattern associated withproduct types, such as product models), the at least one processor maydetect different products and product types. A product of a first typemay be recognized based on a first pattern, and product of a second typemay be recognized based on a second, different pattern (optionally alsobased on weight signal information to aid in differentiating betweenproducts). Such information may be used, for example, to monitor whethera certain region of a shelf includes an appropriate or intended productor product type. Such information may also be useful in determiningwhether products or product types have been mixed (e.g., producthomogeneity). Regarding planogram compliance, detection of differentproducts and their relative locations on a shelf may aid in determiningwhether a product homogeneity value, ratio, etc. has been achieved. Forexample, the at least one processor may count occurrences where aproduct of a second type is placed on an area of the store shelfassociated with a product of a first type.

Additionally or alternatively, the at least one aspect of planogramcompliance may include a restocking rate, and step 1025 may furtherinclude determining the restocking rate based on a sensed rate at whichproducts are added to the at least one area of the store shelfassociated with the second subset of detection elements. Restocking ratemay be determined, for example, by monitoring a rate at which detectionelement signals change as products are added to a shelf (e.g., whenareas of a pressure sensitive pad change from a default value to aproduct-present value).

Additionally or alternatively, the at least one aspect of planogramcompliance may include product facing, and step 1025 may further includedetermining the product facing based on a number of products determinedto be placed on a selected area of the store shelf at a front of thestore shelf. Such product facing may be determined by determining anumber of products along a certain length of a front edge of a storeshelf and determining whether the number of products complies with, forexample, a specified density of products, a specified number ofproducts, and so forth.

Step 1025 may further include transmitting an indicator of the at leastone aspect of planogram compliance to a remote server. For example, asexplained above with respect to FIGS. 8A and 8B, the indicator maycomprise a data packet, a data file, or any other data structureindicating any variations from a planogram, e.g., with respect toproduct (or brand) placement, product facing direction, or the like. Theremote server may include one or more computers associated with a retailstore (e.g., so planogram compliance may be determined on a local basiswithin a particular store), one or more computers associated with aretail store evaluation body (e.g., so planogram compliance may bedetermined across a plurality of retail stores), one or more computersassociated with a product manufacturer, one or more computers associatedwith a supplier (such as supplier 115), one or more computers associatedwith a market research entity (such as market research entity 110), etc.

Method 1000 may further include additional steps. For example, method1000 may include identifying a change in at least one characteristicassociated with one or more of the first signals (e.g., signals from afirst group or type of detection elements), and in response to theidentified change, triggering an acquisition of at least one image ofthe store shelf. The acquisition may be implemented by activating one ormore of capturing devices 125 of FIGS. 4A-4C, as explained above. Forexample, the change in at least one characteristic associated with oneor more of the first signals may be indicative of removal of at leastone product from a location associated with the at least one area of thestore shelf associated with the first subset of detection elements.Accordingly, method 1000 may include triggering the acquisition todetermine whether restocking, reorganizing, or other intervention isrequired, e.g., to improve planogram compliance. Thus, method 1000 mayinclude identifying a change in at least one characteristic associatedwith one or more of the first signals; and in response to the identifiedchange, trigger a product-related task for a store associate of theretail store.

Additionally or alternatively, method 1000 may be combined with method1050 of FIG. 10B, described below, such that step 1055 is performed anytime after step 1005.

FIG. 10B is a flow chart, illustrating an exemplary method 1050 fortriggering image capture of a store shelf, in accordance with thepresently disclosed subject matter. It is contemplated that method 1050may be used in conjunction with any of the detection element arraysdiscussed above with reference to, for example, FIGS. 8A, 8B, and 9 .The order and arrangement of steps in method 1050 is provided forpurposes of illustration. As will be appreciated from this disclosure,modifications may be made to process 1050, for example, adding,combining, removing, and/or rearranging one or more steps of process1050.

Method 1050 may include a step 1055 of determining a change in at leastone characteristic associated with one or more first signals. Forexample, the first signals may have been captured as part of method 1000of FIG. 10A, described above. For example, the first signals may includepressure readings when the plurality of detection elements includespressure sensors, contact information when the plurality of detectionelements includes contact sensors, light readings when the plurality ofdetection elements includes light detectors (for example, from lightdetectors configured to be placed adjacent to (or located on) a surfaceof a store shelf configured to hold products, as described above), andso forth.

Method 1050 may include step 1060 of using the first signals to identifyat least one pattern associated with a product type of the plurality ofproducts. For example, any of the pattern matching techniques describedabove with respect to FIGS. 8A, 8B, and step 1010 may be used foridentification.

Method 1050 may include step 1065 of determining a type of eventassociated with the change. For example, a type of event may include aproduct removal, a product placement, movement of a product, or thelike.

Method 1050 may include step 1070 of triggering an acquisition of atleast one image of the store shelf when the change is associated with afirst event type. For example, a first event type may include removal ofa product, moving of a product, or the like, such that the first eventtype may trigger a product-related task for a store associate of theretail store depending on analysis of the at least one image. Theacquisition may be implemented by activating one or more of capturingdevices 125 of FIGS. 4A-4C, as explained above. In some examples, thetriggered acquisition may include an activation of at least oneprojector (such as projector 632). In some examples, the triggeredacquisition may include acquisition of color images, depth images,stereo images, active stereo images, time of flight images, LIDARimages, RADAR images, and so forth.

Method 1050 may include a step (not shown) of forgoing the acquisitionof at least one image of the store shelf when the change is associatedwith a second event type. For example, a second event type may includereplacement of a removed product by a customer, stocking of a shelf by astore associate, or the like. As another example, a second event typemay include removal, placement, or movement of a product that isdetected within a margin of error of the detection elements and/ordetected within a threshold (e.g., removal of only one or two products;movement of a product by less than 5 cm, 20 cm, or the like; moving of afacing direction by less than 10 degrees; or the like), such that noimage acquisition is required.

FIGS. 11A-11E illustrate example outputs based on data automaticallyderived from machine processing and analysis of images captured inretail store 105 according to disclosed embodiments. FIG. 11Aillustrates an optional output for market research entity 110. FIG. 11Billustrates an optional output for supplier 115. FIGS. 11C and 11Dillustrate optional outputs for store associates of retail store 105.And FIG. 11E illustrates optional outputs for user 120.

FIG. 11A illustrates an example graphical user interface (GUI) 1100 foroutput device 145A, representative of a GUI that may be used by marketresearch entity 110. Consistent with the present disclosure, marketresearch entity 110 may assist supplier 115 and other stakeholders inidentifying emerging trends, launching new products, and/or developingmerchandising and distribution plans across a large number of retailstores 105. By doing so, market research entity 110 may assist supplier115 in growing product presence and maximizing or increasing new productsales. As mentioned above, market research entity 110 may be separatedfrom or part of supplier 115. To successfully launch a new product,supplier 115 may use information about what really happens in retailstore 105. For example, supplier 115 may want to monitor how marketingplans are being executed and to learn what other competitors are doingrelative to certain products or product types. Embodiments of thepresent disclosure may allow market research entity 110 and suppliers115 to continuously monitor product-related activities at retail stores105 (e.g., using system 100 to generate various metrics or informationbased on automated analysis of actual, timely images acquired from theretail stores). For example, in some embodiments, market research entity110 may track how quickly or at what rate new products are introduced toretail store shelves, identify new products introduced by variousentities, assess a supplier's brand presence across different retailstores 105, among many other potential metrics.

In some embodiments, server 135 may provide market research entity 110with information including shelf organization, analysis of skewproductivity trends, and various reports aggregating information onproducts appearing across large numbers of retail stores 105. Forexample, as shown in FIG. 11A, GUI 1100 may include a first display area1102 for showing a percentage of promotion campaign compliance indifferent retail stores 105. GUI 1100 may also include a second displayarea 1104 showing a graph illustrating sales of a certain productrelative to the percentage of out of shelf. GUI 1100 may also include athird display area 1106 showing actual measurements of different factorsrelative to target goals (e.g., planogram compliance, restocking rate,price compliance, and other metrics). The provided information mayenable market research entity 110 to give supplier 115 informed shelvingrecommendations and fine-tune promotional strategies according toin-store marketing trends, to provide store managers with a comparisonof store performances in comparison to a group of retail stores 105 orindustry wide performances, and so forth.

FIG. 11B illustrates an example GUI 1110 for output device 145B used bysupplier 115. Consistent with the present disclosure, server 135 may usedata derived from images captured in a plurality of retail stores 105 torecommend a planogram, which often determines sales success of differentproducts. Using various analytics and planogram productivity measures,server 135 may help supplier 115 to determine an effective planogramwith assurances that most if not all retail stores 105 may execute theplan. For example, the determined planogram may increase the probabilitythat inventory is available for each retail store 105 and may bedesigned to decrease costs or to keep costs within a budget (such asinventory costs, restocking costs, shelf space costs, etc.). Server 135may also provide pricing recommendations based on the goals of supplier115 and other factors. In other words, server 135 may help supplier 115understand how much room to reserve for different products and how tomake them available for favorable sales and profit impact (for example,by choosing the size of the shelf dedicated to a selected product, thelocation of the shelf, the height of the shelf, the neighboringproducts, and so forth). In addition, server 135 may monitor nearreal-time data from retail stores 105 to determine or confirm thatretail stores 105 are compliant with the determined planogram ofsupplier 115. As used herein, the term “near real-time data,” in thecontext of this disclosure, refers to data acquired or generated, etc.,based on sensor readings and other inputs (such as data from imagesensors, audio sensors, pressure sensors, checkout stations, etc.) fromretail store 105 received by system 100 within a predefined period oftime (such as time periods having durations of less than a second, lessthan a minute, less than an hour, less than a day, less than a week,etc.).

In some embodiments, server 135 may generate reports that summarizeperformance of the current assortment and the planogram compliance.These reports may advise supplier 115 of the category and the itemperformance based on individual SKU, sub segments of the category,vendor, and region. In addition, server 135 may provide suggestions orinformation upon which decisions may be made regarding how or when toremove markdowns and when to replace underperforming products. Forexample, as shown in FIG. 11B, GUI 1110 may include a first display area1112 for showing different scores of supplier 115 relative to scoresassociated with its competitors. GUI 1110 may also include a seconddisplay area 1114 showing the market share of each competitor. GUI 1110may also include a third display area 1116 showing retail measurementsand distribution of brands. GUI 1110 may also include a fourth displayarea 1118 showing a suggested planogram. The provided information mayhelp supplier 115 to select preferred planograms based on projected orobserved profitability, etc., and to ensure that retail stores 105 arefollowing the determined planogram.

FIGS. 11C and 11D illustrate example GUIs for output devices 145C, whichmay be used by store associates of retail store 105. FIG. 11C depicts aGUI 1120 for a manager of retail store 105 designed for a desktopcomputer, and FIG. 11D depicts GUI 1130 and 1140 for store staffdesigned for a handheld device. In-store execution is one of thechallenges retail stores 105 have in creating a positive customerexperience. Typical in-store execution may involve dealing with ongoingservice events, such as a cleaning event, a restocking event, arearrangement event, and more. In some embodiments, system 100 mayimprove in-store execution by providing adequate visibility to ensurethat the right products are located at preferred locations on the shelf.For example, using near real-time data (e.g., captured images of storeshelves) server 135 may generate customized online reports. Storemanagers and regional managers, as well as other stakeholders, mayaccess custom dashboards and online reports to see how in-storeconditions (such as, planogram compliance, promotion compliance, pricecompliance, etc.) are affecting sales. This way, system 100 may enablemanagers of retail stores 105 to stay on top of burning issues acrossthe floor and assign store associates to address issues that maynegatively impact the customer experience.

In some embodiments, server 135 may cause real-time automated alertswhen products are out of shelf (or near out of shelf), when pricing isinaccurate, when intended promotions are absent, and/or when there areissues with planogram compliance, among others. In the example shown inFIG. 11C, GUI 1120 may include a first display area 1122 for showing theaverage scores (for certain metrics) of a specific retail store 105 overa selected period of time. GUI 1120 may also include a second displayarea 1124 for showing a map of the specific retail store 105 withreal-time indications of selected in-store execution events that requireattention, and a third display area 1126 for showing a list of theselected in-store execution events that require attention. In anotherexample, shown in FIG. 11D, GUI 1130 may include a first display area1132 for showing a list of notifications or text messages indicatingselected in-store execution events that require attention. Thenotifications or text messages may include a link to an image (or theimage itself) of the specific aisle with the in-store execution event.In another example, shown in FIG. 11D, GUI 1140 may include a firstdisplay area 1142 for showing a display of a video stream captured byoutput device 145C (e.g., a real-time display or a near real-timedisplay) with augmented markings indicting a status of planogramcompliance for each product (e.g., correct place, misplaced, not inplanogram, empty, and so forth). GUI 1140 may also include a seconddisplay area 1144 for showing a summary of the planogram compliance forall the products identified in the video stream captured by outputdevice 145C. Consistent with the present disclosure, server 135 maygenerate within minutes actionable tasks to improve store execution.These tasks may help store associates of retail store 105 to quicklyaddress situations that may negatively impact revenue and customerexperience in the retail store 105.

FIG. 11E illustrates an example GUI 1150 for output device 145D used byan online customer of retail store 105. Traditional online shoppingsystems present online customers with a list of products. Productsselected for purchase may be placed into a virtual shopping cart untilthe customers complete their virtual shopping trip. Virtual shoppingcarts may be examined at any time, and their contents may be edited ordeleted. However, common problems of traditional online shopping systemsarise when the list of products on the website does not correspond withthe actual products on the shelf. For example, an online customer mayorder a favorite cookie brand without knowing that the cookie brand isout-of-stock. Consistent with some embodiments, system 100 may use imagedata acquired by capturing devices 125 to provide the online customerwith a near real-time display of the retail store and a list of theactual products on the shelf based on near real-time data. In oneembodiment, server 135 may select images without occlusions in the fieldof view (e.g., without other customers, carts, etc.) for the nearreal-time display. In one embodiment, server 135 may blur or erasedepictions of customers and other people from the near real-timedisplay. As used herein, the term “near real-time display,” in thecontext of this disclosure, refers to image data captured in retailstore 105 that was obtained by system 100 within a predefined period oftime (such as less than a second, less than a minute, less than about 30minutes, less than an hour, less than 3 hours, or less than 12 hours)from the time the image data was captured.

Consistent with the present disclosure, the near real-time display ofretail store 105 may be presented to the online customer in a mannerenabling easy virtual navigation in retail store 105. For example, asshown in FIG. 11E, GUI 1150 may include a first display area 1152 forshowing the near real-time display and a second display area 1154 forshowing a product list including products identified in the nearreal-time display. In some embodiments, first display area 1152 mayinclude different GUI features (e.g., tabs 1156) associated withdifferent locations or departments of retail store 105. By selectingeach of the GUI features, the online customer may virtually jump todifferent locations or departments in retail store 105. For example,upon selecting the “bakery” tab, GUI 1150 may present a near real-timedisplay of the bakery of retail store 105. In addition, first displayarea 1152 may include one or more navigational features (e.g., arrows1158A and 1158B) for enabling the online customer to virtually movewithin a selected department and/or virtually walk through retail store105. Server 135 may be configured to update the near real-time displayand the product list upon determining that the online customer wants tovirtually move within retail store 105. For example, after identifying aselection of arrow 1158B, server 135 may present a different section ofthe dairy department and may update the product list accordingly. Inanother example, server 135 may update the near-real time display andthe product list in response to new captured images and new informationreceived from retail store 105. Using GUI 1150, the online customer mayhave the closest shopping experience without actually being in retailstore 105. For example, an online customer can visit the vegetabledepartment and decide not to buy tomatoes after seeing that they are notripe enough.

It has become more common for customers to order goods from local retailstores for home delivery, including but not limited to orderinggroceries from a local grocery store for home delivery. In the exemplarycase of groceries, the goods are often perishable and must be deliveredin a timely manner from a retail store near the customer to ensuredelivery without product spoilage due to time spent out of aclimate-controlled environment (e.g., a refrigerator or freezer).

When a customer places an order for goods from a retail store, usuallythrough an online ordering application or website, the order may becommunicated as a customer shopping list to a shopper located in or nearthe retail store. The shopper is typically, though not necessarily, anindividual human being. The customer's shopping list will contain a listof at least one target product that the customer is interested in havingdelivered from the retail store. In many cases, a target product will bespecified by the customer as being a particular brand, size, weight,quantity, packaging etc. The shopper receiving the shopping list is thentasked with efficiently collecting target products on the shopping listfrom the retail store inventory in accordance with the shopping listspecifications and then taking steps for the checkout and timelydelivery to the customer of the specified target items.

The pace of store pickers (i.e., an in-store “shopper”) is an importantfactor to the operational efficiency of retailers. As used herein, theterm “store picker” and “shopper” may be used interchangeably, unlessotherwise expressly indicated, to refer to an in-store individual who isretrieving products from in-store inventory for checkout and deliveryinto the possession and/or ownership of a customer. As used herein, theterm “in-store” generally means to be physically located in or at astore. In some embodiments, it is contemplated that the shopper andcustomer may be the same. A main challenge to shoppers is out-of-stockitems. Dealing with out-of-stock items is both time consuming andaffects the quality of service to the end customer. Supporting in-storeshoppers when an item is out-of-stock may therefore increase theefficiency of the shopper. Further, it may improve customer satisfactionby enabling the shopper to select more suitable replacement items.Therefore, suggesting alternative items for out-of-stock items in anatural and efficient way is desired. Described herein is a system forassisting in-store shoppers in locating an alternative product when atarget product on a customer shopping list is unavailable in-store(i.e., is not part of the store's physical inventory at the time ofshopping).

FIG. 12 illustrates an exemplary embodiment of system 100 for locatingan available alternative product in store 105 c that includes at leastone computer processor configured to execute one or more operatinginstructions of a computer program located in a non-transitory computerreadable medium. As depicted in FIG. 12 , server 135 may becommunicatively coupled to one or more physical or virtual storagedevices such as database 140. Server 135 may access database 140 todetect and/or identify products. The detection may occur throughanalysis of features in a store image using an algorithm and storeddata.

As used herein, the term “store image” includes any data representativeof the physical appearance, contents, and/or occupancy of at least aportion of the physical environment of store at a given moment in time,or during a given time span. For example, a store image may be capturedby an in-store camera of shelf 1230. In an example embodiment, thesystem may capture the at least one store image by an image sensorincluded on a mobile device associated with a shopper. By way ofexample, a mobile device associated with a shopper may be correspond todevices 1210 or 1220 illustrated in FIG. 12 . In another exampleembodiment, the at least one store image may be captured by a cameramounted in the particular retail store as part of the storeinfrastructure, such as the camera device 125 shown in FIG. 12 .

The disclosed system may identify a product by analyzing productfeatures in an image relative to one or more product models. Consistentwith the present embodiment, the term “product model” refers to any typeof algorithm or stored data that a processor may access or execute toenable the identification of a particular product associated with theproduct model. For example, the product model may include a descriptionof visual and contextual properties of the particular product (e.g., theshape, the size, the colors, the texture, the brand name, the price, thelogo, text appearing on the particular product, the shelf associatedwith the particular product, adjacent products in a planogram, thelocation within the retail store, etc.). In some embodiments, a singleproduct model may be used by server 135 to identify more than one typeof product, such as, when two or more product models are used incombination to enable identification of a product. For example, in somecases, server 135 may use a first product model to identify a productcategory (such models may apply to multiple product types, e.g.,shampoo, soft drinks, etc.), and server 135 may use a second productmodel to identify the product type, product identity, or othercharacteristics associated with a product. In some cases, server 135 mayapply the product models together (e.g., in series, in parallel, in acascade fashion, in a decision tree fashion, etc.) to reach a productidentification. In other embodiments, server 135 may use a singleproduct model to identify a particular product type (e.g., 6-pack of 16oz Coca-Cola Zero).

Database 140 may be included on a volatile or non-volatile, magnetic,semiconductor, tape, optical, removable, non-removable, or other type ofstorage device or tangible or non-transitory computer-readable medium.Database 140 may also be part of server 135 or separate from server 135.When database 140 is not part of server 135, server 135 may exchangedata with database 140 via a communication link. Database 140 mayinclude one or more memory devices that store data and instructions usedto perform one or more features of the disclosed embodiments. In oneembodiment, database 140 may include one or more suitable databases,ranging from small databases hosted on a workstation to large databasesdistributed among data centers. Database 140 may also include anycombination of one or more databases controlled by memory controllerdevices (e.g., server(s), etc.) or software. For example, database 140may include document management systems, Microsoft SQL databases,SharePoint databases, Oracle™ databases, Sybase™ databases, otherrelational databases, or non-relational databases, such as mongo andothers.

Consistent with the present disclosure, image processing unit 130 maycommunicate with a mobile device that is associated with a shopper. Insome embodiments, the mobile device associated with the shopper is anextended reality appliance. FIG. 12 shows an exemplary embodiment ofwearable extended reality appliance 1210. FIG. 12 also illustrates analternative exemplary embodiment of hand-held extended reality appliance1220. Whether wearable or hand-held, it is contemplated that an extendedreality appliance will at least be a network device capable ofcommunicating with a server (e.g., server 135) over one or more networks(e.g., via communications network 150).

The term “extended reality appliance” as used herein includes a devicecapable of generating and/or overlaying computer-generated or digitaltext, graphics, images, videos, animations, or three-dimensionalholograms over a real-world scene or environment seen by a user from thepoint of view of the user. By way of example, an extended realityappliance may have a “heads-up” transparent display that presents datato the user's screen in front of the user's eyes, so that the user neednot look away from their usual viewpoints. By way of another example, anextended reality appliance may display 3D holograms overlaid on the realworld where the user is located to render a mixed reality experience tothe user. The hologram image may be generated using light diffractiontechniques. By way of another example, an extended reality appliance maybe a head-worn device in the form of glasses or goggles, such asextended reality appliance 1210 illustrated in FIG. 12 . By way of yetanother example, an extended reality appliance may be a hand-helddevice, such as a mobile device, as shown for extended reality appliance1220 illustrated in FIG. 12 .

It is contemplated that an extended reality appliance of the disclosedsystem may be a dedicated and exclusive system component, oralternatively may be a multi-purpose device having features (i.e.,hardware) and/or applications (i.e., software) that make it suitable fortemporary use of the device in the system by a shopper as an extendedreality appliance. For example, in a situation where the shopper is thecustomer, the personal mobile communications device of the shopper thatis used as a part of the system may also be capable of being used as atemporary extended reality appliance (“TERA”) component of the systemfor the time span that the shopper is in-store shopping for products.Such TERA functionality may be achieved, for example, by the customerdownloading and installing onto a personal mobile communications devicea temporary extended reality appliance software (“TERAS”) applicationfor use when shopping in-store.

An extended reality appliance may also be a dedicated system device thatuses a common multi-purpose hardware platform specially configured forexclusive use in the system. Such an exclusive system extended realityappliance (“ESERA”) may, for example, be in the form of a commonhardware platform such as a mobile phone (e.g., an Apple iPhone, GooglePixel, Samsung Galaxy, etc.) or tablet device (e.g., an Apple iPad,Google Pixel Slate, Samsung Galaxy Tab, etc.). Such common hardwareplatform may be used as an ESERA by running an extended realityappliance software (“ERAS”) application that is configured to only allowthe device to be used with the system. It is contemplated that suchESERA devices may be provided to individuals who are employed orcontracted as shoppers on the system.

In an example embodiment of the described system, a processor of server135 executes one or more operating instructions located in anon-transitory computer readable medium to obtain an electronic shoppinglist associated with a customer, with the electronic shopping listidentifying a target product. As used herein, the term “shopping list”means any data received from a customer identifying at least one targetproduct. As used herein, a “target product” is any product specificallyidentified by a customer for acquisition from a particular store. In anexample embodiment, the customer shopping list is obtained by the systemover communications network 150 after being transmitted from customer120 mobile device 145D.

In an example embodiment, the processor of server 135 may execute one ormore operating instructions located in a non-transitory computerreadable medium to receive at least one image representative of at leasta portion of the particular retail store (i.e., a store image). The atleast one image may include a representation of an alternative product.As used herein, the term “alternative product” means a product having asimilar function and/or characteristics as a target product such that itmay be considered a suitable replacement or substitute for the targetproduct. In another example embodiment, the at least one image is aframe of a video feed. In another example embodiment, the at least oneimage includes a series of images. In such an embodiment, the series ofimages may be analyzed to determine whether the alternative product isselected by the shopper. After it is determined that the alternativeproduct type has been selected by the shopper, the system mayautomatically update the shopping list to reflect a selection of thealternative product type by the shopper, and/or send a notification tothe customer indicating a selection of the alternative product type bythe shopper.

A processor of server 135 may execute one or more operating instructionslocated in a non-transitory computer readable medium to automaticallydetermine that the target product is unavailable at a particular retailstore. For example, a processor of server 135 may execute one or moreoperating instructions located in a non-transitory computer readablemedium to automatically determine that the target product is unavailableat the particular retail store based on an analysis of the at least oneimage. For example, the at least one image may include a representationof a location in the retail store (e.g., such as a portion of aparticular retail shelf) where the target product is supposed to be at,for example, based on a store map, based on a planogram, based on arealogram, based on analysis of previous images showing the targetproduct at that location, based on a sign identifying the target product(e.g., textually, through a barcode or another visual code, by includingan image of the target product, etc.) and corresponding to that location(e.g., a sign attached to the portion of the particular retail shelf),and so forth. Further, the at least one image may be analyzed todetermine that no items of the target product are available at thatlocation, for example using product recognition algorithm, and therebydetermine that the target product is unavailable at the particularretail store. As another example, a processor of server 135 may executeone or more operating instructions located in a non-transitory computerreadable medium to automatically determine that the target product isunavailable at the particular retail store based on analysis ofinventory data associated with the particular retail store.

In some embodiments, a processor of server 135 may execute one or moreoperating instructions located in a non-transitory computer readablemedium to automatically identify an alternative product, available atthe particular retail store, as a potential alternative to the targetproduct. For example, a processor of server 135 may execute one or moreoperating instructions located in a non-transitory computer readablemedium to automatically identify an alternative product, available atthe particular retail store, based on an analysis of the at least oneimage. In some examples, the at least one image may be analyzed usingproduct recognition algorithms to identify products available at theparticular retail store, and the alternative product may be selectedfrom the identified products available at the particular retail store.For example, affinities (e.g., based on similarity among products, basedon dissimilarities among products, for example based on characteristicsof products obtained from a database and/or from an analysis of the atleast one image) may be used to select as the alternative product theproduct with the highest affinity to the target products among theidentified products available at the particular retail store. In anotherexample, the at least one image may be analyzed to determine a quantityof items for each one of a plurality of products (e.g., using aregression model, using object counting algorithms, etc.), and aselection of a particular product as the alternative product may beavoided when the quantity of items corresponding to that product isbelow a selected threshold. The threshold may be different for differentproducts or may be the same to all products. In yet another example, theat least one image may be analyzed to determine a price for each one ofa plurality of products (e.g., by analyzing price labels using OCRalgorithms), a sensitivity to price of the customer may be obtained(e.g., from the customer, from an analysis of previous purchases of thecustomer, etc.), and the alternative product may be selected based onthe determined prices and the obtained sensitivity to price of thecustomer (e.g., avoiding selection of high priced products for acustomer with high sensitivity to price, while selection a high pricedproduct for a customer with low sensitivity to price). In otherembodiments, a processor of server 135 may execute one or more operatinginstructions located in a non-transitory computer readable medium toautomatically identify an alternative product, available at theparticular retail store, based on analysis of inventory data associatedwith the particular retail store. For example, a selection of aparticular product as the alternative product may be avoided when thequantity of items corresponding to that product is below a selectedthreshold. The threshold may be different for different products or maybe the same to all products.

A processor of server 135 may execute one or more operating instructionslocated in a non-transitory computer readable medium to analyze the atleast one image to cause an extended reality appliance to present to ashopper a visual indicator associated with the alternative product inthe particular retail store. For example, the at least one image may beanalyzed to determine a position (e.g., such as location, orientation,etc.) in the physical retail store to present the virtual visualindicator at. A digital signal may be provided to the extended realityappliance (e.g., through a digital communication device) to cause theextended reality appliance to present the visual indicator at thedetermined position. In one example, the digital signal may include anindication of the position (e.g., in a coordinate system relative to atleast one of the world, the retail store, one or more objects within theretail store, or the extended reality appliance (for example, at aparticular point in time). In another example, the at least one imagemay be analyzed to select of the visual indicator or of a characteristicof the visual indicator (e.g., such as size, color scheme, opacity,graphical content, textual content, and so forth). A digital signal maybe provided to the extended reality appliance (e.g., through a digitalcommunication device) to cause the extended reality appliance to presentthe visual indicator with the selected characteristic. For example, thedigital signal may include a digital encoding of the selectedcharacteristic. In some examples, a machine learning model may betrained using training examples to determine information associated withvisual indicators (e.g., such as position information, visual indicatorcharacteristic information, etc.) from images and/or videos. An exampleof such training example may include a sample image and/or a samplevideo, together with a label indicating the desired determination ofinformation for the sample image and/or the sample video. The trainedmachine learning model may be used to analyze the at least one image todetermine information associated with a visual indicator associated withthe alternative product. Further a digital signal based on the determineinformation associated with a visual indicator associated with thealternative product may be provided to the extended reality appliance(e.g., through a digital communication device) to cause the extendedreality appliance to present the visual indicator. For example, thedigital signal may include a digital encoding of the determinedinformation associated with a visual indicator associated with thealternative product. In some embodiments, extended reality appliance maybe a head-mounted device as shown for example by device 1210 in FIG. 12. In other embodiments, the extended reality appliance may be a mobilecommunication device associated with the shopper, as shown for exampleby device 1220 in FIG. 12 .

A processor of server 135 may execute one or more operating instructionslocated in a non-transitory computer readable medium to cause a visualindicator associated with the alternative product in the particularretail store to be displayed to a shopper on an extended realityappliance. For example, the extended reality appliance may include asee-through display 1222, such as shown in FIG. 13B, and the analysis ofthe at least one image may be used to select a location on thesee-through display 1222 so that a visual indicator is positioned in acorrect location with respect to the alternative product.

In some embodiments, the visual indicator may be presented as a firstindicator type if substitution of the target product with thealternative product requires an authorization from the customer, andwherein the visual indicator may be presented as a second indicatortype, different from the first indicator type, if substitution of thetarget product with the alternative product does not require anauthorization from the customer. FIGS. 13A and 13C show examples ofvisual indicators. Specifically, with reference to FIG. 13A, wearableextended reality appliance 1210 includes a display screen 1310 and atleast a portion of the at least one store image 1320 is displayed ondisplay screen 1310 together with a graphic overlay 1312 comprised ofvarious informational elements (1312A, 1312B, 1312C and 1312D),including visual indicator 1312A associated with a representation ofalternative product 1311. As illustrated in FIG. 13A visual indicator1312A is solid black text with no background. FIG. 13C shows a hand-heldextended reality appliance 1220 having a display screen 1350. Asillustrated in FIG. 13C, at least a portion of the at least one storeimage 1360 is displayed on display screen 1350 together with a graphicoverlay 1352 including various informational elements (1352A, 1352B,1352C and 1352D and visual indicator 1352A, which is associated with arepresentation of alternative product 1361. As illustrated in FIG. 13D,visual indicator 1352A is solid white text with a black background.Visual indicator 1312A and visual indicator 1352A are examples ofdifferent visual indicator types for communicating whether customerauthorization is needed to select the identified alternative product.For example, a visual indicator with a solid black text and nobackground like visual indicator 1312A may represent a first type ofvisual indicator communicating that substitution of the target productwith the alternative product requires customer authorization, while avisual indicator with a white text on a solid black background like thevisual indicator 1352A may represent a second type of visual indicatorcommunicating that substitution of the target product with thealternative product does not require customer authorization. Thepreceding are merely examples and in other embodiments, the visualindicators may differ from the illustrated examples and includegraphical symbols, animated objects, etc.

As shown in FIG. 13A, a graphic overlay 1312 displayed on display 1300of extended reality appliance 1210 may also include a visual indication1312D that the target product is not available at the particular retailstore. Similarly, referring to the example of FIG. 13C, a graphicoverlay 1352 displayed on display 1350 of an extended reality appliance1220 may also include a visual indication 1352D that the target productis not available at the particular retail store.

As shown in FIG. 13A, a graphic overlay 1312 displayed on display 1300of extended reality appliance 1210 may also include a directionalindicator 1312C for assisting in locating the alternative product withinthe particular retail store. Similarly, referring to the example of FIG.13C, a graphic overlay 1352 displayed on display 1350 of an extendedreality appliance 1220 may also include a directional indicator 1352Cfor assisting in locating the alternative product within the particularretail store. For example, in an embodiment where the shopper isdisplaced from the location of the alternative product, the system maycapture an image of the alternative product and display the capturedimage to the shopper along with directions, directional arrows, etc. forlocating the alternative product.

As shown in FIG. 13A, a graphic overlay 1312 displayed on display 1300of extended reality appliance 1210 may also include an indication 1312Bof a location in the particular retail store where the alternativeproduct is located. Similarly, referring to the example of FIG. 13C, agraphic overlay 1352 displayed on display 1350 of an extended realityappliance 1220 may also include an indication 1352B of a location in theparticular retail store where the alternative product is located.

FIG. 14A includes a flowchart representing an exemplary method 1400A forlocating an available alternative product in a store 105 c in accordancewith example embodiments of the present disclosure. For purposes ofillustration, in the following description, reference is made to certaincomponents of system 100 as deployed in the configuration depicted inFIGS. 12 and 13A-C. For example, method 1400A may be performed by atleast one processor or at least one processing unit, such as processingunit 130 or processing device 202. It is to be understood thatthroughout the present disclosure, the term “processor” is used as ashorthand for “at least one processor.” In other words, a processor mayinclude one or more structures that perform logic operations whethersuch structures are collocated, connected, or dispersed. In someembodiments, a non-transitory computer readable medium may containinstructions that when executed by a processor cause the processor toperform method 1400A. Further, method 1400A is not necessarily limitedto the steps shown in FIG. 14A, and any steps or processes of thevarious embodiments described throughout the present disclosure may alsobe included in method 1400A.

At step 1402A, method 1400A includes a processor of server 135 executingone or more operating instructions from a computer program located in anon-transitory computer-readable medium to obtain a shopping list overcommunications network 150 from user 120, where the shopping listidentifies at least one target product for selection from the store 105c. The shopping list may be entered by a customer user of the systemthrough a personal mobile communications device associated with thecustomer user.

At step 1404A, one or more operating instructions of the computerprogram are executed by a processor of server 135 to automaticallydetermine the at least one target product as being an unavailableproduct at store 105 c. Such an automatic determination may initially bedone on receipt of the communicated shipping list by making a comparisonto the most recent store inventory records in an electronic inventorydatabase.

At step 1406A, one or more operating instructions of the computerprogram are executed by a processor of server 135 to automaticallyidentify an available alternative product at the store 105 c. Such anautomatic determination may initially be done on identification of atarget product being unavailable by querying the electronic inventorydatabase for an identification of available products that are consideredalternatives to the target product.

At step 1408A, one or more operating instructions of the computerprogram are executed by a processor of server 135 to receive at leastone image from the store 105 c including a representation of thealternative product. The received images may be from image capturedevices that are part of the store infrastructure, such as wall mountedcameras, or may be mobile image capture devices, such as from devicescarried by in-store shoppers or autonomous in-store mobile image capturedevices like a drone.

At step 1410A, one or more operating instructions of the computerprogram are executed by a processor of server 135 to analyze the atleast one image received from the store 105 c. The at least one storeimage may be a captured still image, or a video frame, or a series ofstill images or video frames. The analysis of the at least one storeimage may be used to automatically identify a target product as beingout of stock, such as through an image analysis identifying empty storeshelf space where the target product would be expected to be locatedaccording to a store planogram. The availability of alternative productscould be confirmed through analysis of a store image, or other sensorinput, of the presence of the alternative product on a store shelf.

At step 1412A, one or more operating instructions of the computerprogram are executed by a processor of server 135 to display on anextended reality appliance 1210 a visual indicator 1312A of thealternative product at the store 105 c. The visual indicator may becomprised of text and/or graphics identifying and showing the locationand availability of the alternative product.

FIG. 14B includes a flowchart representing an exemplary method 1400B forlocating an available alternative product in a store 105 c in accordancewith example embodiments of the present disclosure. For purposes ofillustration, in the following description, reference is made to certaincomponents of system 100 as deployed in the configuration depicted inFIGS. 12 and 12A-D. For example, method 1400B may be performed by atleast one processor or at least one processing unit, such as processingunit 130 or processing device 202. It is to be understood thatthroughout the present disclosure, the term “processor” is used as ashorthand for “at least one processor.” In other words, a processor mayinclude one or more structures that perform logic operations whethersuch structures are collocated, connected, or dispersed. In someembodiments, a non-transitory computer readable medium may containinstructions that when executed by a processor cause the processor toperform method 1400B. Further, method 1400B is not necessarily limitedto the steps shown in FIG. 14B, and any steps or processes of thevarious embodiments described throughout the present disclosure may alsobe included in method 1400B.

At step 1402B, method 1400B includes a processor of server 135 executingone or more operating instructions from a computer program located in anon-transitory computer-readable medium to obtain a shopping list overcommunications network 150 from user 120, where the shopping listidentifies at least one target product for selection from the store 105c. The shopping list may be entered by a customer user of the systemthrough a personal mobile communications device associated with thecustomer user.

At step 1404B, one or more operating instructions of the computerprogram are executed by a processor of server 135 to automaticallydetermine the at least one target product as being an unavailableproduct at store 105 c. Such an automatic determination may initially bedone on receipt of the communicated shipping list by making a comparisonto the most recent store inventory records in an electronic inventorydatabase.

At step 1406B, one or more operating instructions of the computerprogram are executed by a processor of server 135 to automaticallyidentify an available alternative product at the store 105 c. Such anautomatic determination may initially be done on identification of atarget product being unavailable by querying the electronic inventorydatabase for an identification of available products that are consideredalternatives to the target product.

At step 1408B, one or more operating instructions of the computerprogram are executed by a processor of server 135 to receive at leastone image from the store 105 c including a representation of thealternative product. The received images may be from image capturedevices that are part of the store infrastructure, such as wall mountedcameras, or may be mobile image capture devices, such as from devicescarried by in-store shoppers or autonomous in-store mobile image capturedevices like a drone.

At step 1410B, one or more operating instructions of the computerprogram are executed by a processor of server 135 to analyze the atleast one image received from the store 105 c. The at least one storeimage may be a captured still image, or a video frame, or a series ofstill images or video frames. The analysis of the at least one storeimage may be used to automatically identify a target product as beingout of stock, such as through an image analysis identifying empty storeshelf space where the target product would be expected to be locatedaccording to a store planogram. The availability of alternative productscould be confirmed through analysis of a store image, or other sensorinput, of the presence of the alternative product on a store shelf.

At step 1412B, one or more operating instructions of the computerprogram are executed by a processor of server 135 to determine ifsubstitution of the in-store available alternative product as areplacement for the unavailable target product requires customerauthorization. A customer user of the system, when inputting orcommunicating their shopping list, may provide an authorizationindication or setting as to whether customer authorization is requiredfor the substitution of a target product on the shopping list with anavailable alternative product. The authorization indication or settingmay be stored by the system and used by the system when an alternativeproduct is available for substitution.

At conditional step 1414B, one or more operating instructions of thecomputer program may be executed by a processor of server 135 to displayon an extended reality appliance a first alternative product visualindicator type if it was determined at step 1412B that customerauthorization was needed for substitution. The system may display such afirst visual indicator in a particular visual style, such as plain text,or text of a certain color. At conditional step 1416B, one or moreoperating instructions of the computer program may be executed by aprocessor of server 135 to display on an extended reality appliance asecond alternative product visual indicator type, distinctive inappearance from the first alternative product visual indicator type, ifit was determined at step 1412B that customer authorization was notneeded for substitution. The second alternative product visual indicatortype should be readily distinguishable in appearance from the firstvisual indicator type, such as for example having a prominentcontrasting background.

At step 1418B, one or more operating instructions of the computerprogram are executed by a processor of server 135 to determine if thealternative product has been selected by the shopper. This may beaccomplished through the shopper taking an action through the extendedreality appliance to indicate selection of the alternative product, orcould be determined automatically through image analysis or sensor inputshowing a change in the shelf inventory of the alternative product.

At step 1420B, one or more operating instructions of the computerprogram are executed by a processor of server 135 to update the shoppinglist based on whether the alternative product was selected or not by theshopper.

With the increased use of internet-connected mobile communicationdevices by in-store shoppers there is also an increased opportunity fora store to still satisfy a customer's demand for an in-store productwhen the shopper encounters an out-of-stock situation at a store. Thesystem and methods described herein provide for the automatic sensing ofsuch a situation. Specifically, to facilitate online shopping forcustomers in retail stores, the system may receive indicators when acustomer who is physically present in a retail store is interested in acategory of products. The system may also receive indicators when aproduct targeted by the customer is out of stock. In response to theseindicators, the system may send a notification to the customersuggesting purchasing the product online, e.g., via a mixed realityplatform. For example, a customer may wish to buy a specific brand ofdetergent that is currently unavailable: Rather than purchase adifferent brand, the system may suggest that the customer purchase thedetergent online, e.g., to preserve brand loyalty.

The system described herein, as illustrated by way of example in FIG. 12, may also provide an in-store shopper notification about the onlineordering of an unavailable in-store product. Such a use of the systemmay apply to circumstances where the in-store shopper is also thecustomer.

FIGS. 15A and 15B illustrate an exemplary head-worn extended realityappliance 1210 and a display 1300 for extended reality appliance 1210.In an example embodiment, one or more operating instructions included ina non-transitory computer readable medium are executed by a processor ofserver 135 to present on display 1300 a contemporaneous store image fromat least a portion of the store, such as a shelf space, which includes arepresentation of an unavailable in-store product 1510.

In an example embodiment, one or more operating instructions included ina non-transitory computer readable medium are executed by a processor ofserver 135 to make a first determination that a customer located in aphysical retail store has engaged in at least one product categoryaction associated with an interest in a category of products associatedwith the retail store. As used herein, the term “product categoryaction” means any action of a shopper that is associated with a categoryof products (i.e., a product category). As used herein, the term“product category” means a collection of products that share a commonfunction and/or characteristics. In some embodiments, one or moreoperating instructions included in a non-transitory computer readablemedium are executed by a processor of server 135 to make the firstdetermination based on automatic analysis of image data acquired by oneor more cameras located in the physical retail store, where in someembodiments the one or more cameras are mounted within the physicalretail store, and in other embodiments the one or more cameras areincluded on a device associated with the shopper. The device associatedwith the shopper may include a mobile device, a phone, glasses, etc. Insome examples, the image data may capture the at least one actionassociated with the interest in the category of products associated withthe retail store, the image data may be analyzed using an actionrecognition algorithm to detect the at least one action, and the firstdetermination may be based on the detection of the at least one action.Some non-limiting examples of such actions for associated with theinterest in the category of products may include picking up at least oneproduct from among the category of products, reaching to at least oneproduct from among the category of products, looking in a direction ofat least one product from among the category of products, and so forth.In some examples, a machine learning model may be trained using trainingexamples to determine whether customers are engaged in actionsassociated with interest in categories of products from images and/orvideos. An example of such training example may include a sample imageand/or a sample video of a sample customer in a sample retail store,together with a label indicating whether the sample customer is engagedin an action associated with an interest in a particular category ofproducts associated with the sample retail store. The trained machinelearning model may be used to analyze the image data and make the firstdetermination. In some examples, a convolution of at least part of theimage data may be calculated to obtain a result value of the calculatedconvolution. In one example, the first determination may be based on theresult value of the calculated convolution. In one example, in responseto the result value of the calculated convolution being a first value,the first determination may be made, and in response to the result valueof the calculated convolution being a second value, making the firstdetermination may be avoided.

In an example embodiment, one or more operating instructions included ina non-transitory computer readable medium are executed by a processor ofserver 135 to make the first determination based at least in part on asensed location of the customer within the physical retail store.Extended reality appliance 1210, either alone or in conjunction within-store infrastructure, may communicate to server 135 in-store shopperlocation information from which a first determination may be made of aproduct category action for the shopper. It is common in retail storeenvironments for products that share a common function and/orcharacteristics to be grouped together at a particular in-storelocation. For example, paper products like toilet paper, tissue, andabsorbent towels are typically displayed on adjacent shelves in aparticular aisle of a store, while bread and other baked goods aretypically displayed on adjacent shelves in a different aisle of thestore. Accordingly, physical information about a shopper's in-storelocation may be collected and analyzed by the system in accordance withone or more operating instructions to make a first determination aboutthe product category actions of an in-store shopper.

In an example embodiment, one or more operating instructions included ina non-transitory computer readable medium may also be executed by aprocessor of server 135 to make the first determination based at leastin part on a looking direction (i.e., a viewing direction) 1530 of anin-store shopper 1505. The viewing direction 1530 of the shopper 1505may be sensed based on an analysis of a image data acquired by one ormore cameras associated with the physical retail store. For example, theimage data may be analyzed using a gaze detection algorithm and/or usinghead pose estimation algorithm to determine viewing direction 1530. Theview direction may also be determined from the sensors of an extendedreality appliance, such as a magnetometer to provide compass readings(i.e., horizontal direction), and one or more accelerometers for levelreadings (i.e., vertical direction), for the extended reality appliance.

In another an example embodiment, one or more operating instructionsincluded in a non-transitory computer readable medium may be executed bya processor of server 135 to make the first determination based at leastin part on an amount of time the looking direction of the customerremains within a certain range of looking directions (i.e., a viewdirection time 1540). The view direction time may be determined by imageanalysis of the shopper in-store, or could also be determined from acombination of a clock and sensors of an extended reality appliance,where the time could be recorded which was spent by the extended realityappliance in a particular range of horizontal and vertical orientationas determined by a magnetometer to provide compass readings (i.e.,horizontal direction), and one or more accelerometers for level readings(i.e., vertical direction), for the extended reality appliance.

Referring to FIG. 15C, in another example embodiment, one or moreoperating instructions included in a non-transitory computer readablemedium may be executed by a processor of server 135 to make the firstdetermination based at least in part on a sensed action by the customerrelative to the category of products associated with the physical retailstore, such as for example picking up at least one product 1515 fromamong the category of products. The determination of the picking up ofat least one product may be determined through analysis of at least oneimage captured of the in-store shopper performing such an action, and/oranalysis of sensor data regarding the presence or absence of productfrom a store shelf correlated with the proximity of the shopper to suchstore shelf.

In another example embodiment, one or more operating instructionsincluded in a non-transitory computer readable medium may be executed bya processor of server 135 to make the first determination based at leastin part on an automatic analysis of a virtual shopping list associatedwith the customer. A virtual shopping list may be a shopping list that acustomer user of the system enters into a mobile communications deviceassociated with the customer and communicates to the system server 135.The virtual shopping list identifies at least one target product thatthe customer user is interested in obtaining from the store, and thesystem may execute one or more operating instructions to analyze suchshopping list to determine that a shopper located in the physical retailstore has an interest in a target product that is part of a productcategory associated with the retail store. Such analysis may include anidentification of the product category of a target product throughreference to one or more inventory databases.

In another example embodiment, one or more operating instructionsincluded in a non-transitory computer readable medium may be executed bya processor of server 135 to make the first determination based at leastin part on an analysis of a previous shopping activity associated withthe shopper. The system disclosed herein may store data associated witha shopper's prior shopping activity by maintaining data from a shopper'sactivities on previous visits to the physical retail store (e.g., whatthe shopping list was, and what was actually purchased, on such priorvisits), or from prior activity of the shopper during a particular storevisit, and execute one or more operating instructions to use such datain making a determination that at least one action of the shopper isassociated with an interest in a product category.

Accordingly, physical information about a shopper's in-store location,shopper's view direction 145 (i.e., the direction the shopper islooking), the shopper's view direction time 146 (i.e., how much time theshopper spends looking in a particular direction), and the products theshopper is picking up and/or otherwise inspecting are examples ofphysical information the disclosed system may collect and analyze tomake a first determination about the product category actions of anin-store shopper. The system may obtain this type of contemporaneousshopper physical information, in whole or in part, from in-storeinfrastructure such as in-store imaging cameras or other sensingequipment (e.g., RADAR, LIDAR, pressure sensors, acoustic sensors,etc.,) that may be mounted on in-store walls, ceilings, shelves,shopping carts etc. Such physical information can also be obtained, inwhole or in part, from devices that are associated with the in-storeshopper, such as a hand-held mobile communication device (e.g., a cellphone compass and/or level) or a wearable device such as an extendedreality appliance headset or a smart watch using embedded technologiessuch as cameras, clocks, accelerometers, compasses (magnetometers), GPSpositioning, Bluetooth, infra-red, RFID, and location tracking. Thesystem may, for example, use server 135 or a shopper device 1210 or1220, to execute one or more operating instructions to determine atleast one product category action by shopper based at least in part onan analysis of a store image 1520 captured by in-store infrastructure ora device associated with the shopper. For example, an image may becaptured of the shelf space in the view direction of the shopper, andthe system may analyze the captured image for empty shelf space showingthat an in-store product is unavailable (i.e., is an out-of-stock itemin-store). In addition to making the first determination of shopperproduct category actions from information about shopper physical actionsand the shopper location or environment, non-physical information suchas prior shopping activity and a shopping list associated with theshopper may also be used in making such product category actiondeterminations.

The disclosed system may use in-store location information, viewdirection information, and/or view direction time information co make afirst determination that at least one product category action has beentaken by the in-store shopper. For example, by sensing where in a storeaisle the shopper is located, and which shelf space is within thedirection of view for the shopper, and how long the shopper hasmaintained the direction of view, the system can make a firstdetermination that the in-store shopper is presently in the vicinity ofand/or focused on a particular product category (i.e., certain types ofproducts) and is taking at least one product category action. By way ofexample, a first product category action may be defined in the system asthe in-store shopper having located themselves at a particular in-storeproduct category location (e.g., a particular store aisle and shelfspace), and had a particular view direction for at least a thresholdview direction time. By way of another example, a second productcategory action may be defined by the shopper having located themselvesat a particular in-store product category location (e.g., a particularstore aisle and shelf space), had a particular view direction, andpicked up an in-store product from a shelf. By way of yet anotherexample, a third product category action may be defined by the shopperhaving located themselves at a particular in-store product categorylocation (e.g., a particular store aisle and shelf space) and have anassociated shopping list including a target product from the productcategory. There is no limit on the different number of contemporaneousphysical information, historical shopper patterns, or shopper datainput, combinations that may be used by the system in making the firstdetermination of a shopper taking a product category action.

In an exemplary embodiment, once a first determination has been made ofa shopper taking at least one product category action, the system mayexecute one or more operating instructions to make a seconddetermination that the at least one product category action isassociated with a presently unavailable (i.e., out-of-stock) in-storeproduct. For example, one or more operating instructions included in anon-transitory computer readable medium may be executed by a processorof server 135 to make a second determination that the at least oneaction is associated with a target product, included in the category ofproducts, that is out of stock in the physical retail store. The seconddetermination that a shopper action in a product category is associatedwith a target product that is out of stock may be based on an analysisof the target products on a customer shopping list in combination withphysical information about a shopper's in-store location, the shopper'sview direction (i.e., the direction the shopper is looking), theshopper's view direction time (i.e., how much time the shopper spendslooking in a particular direction), and the products the shopper ispicking up and/or otherwise inspecting. In some embodiments the systemmay provide out-of-stock alert devices in proximity to the shelf-spacethat a product ordinarily occupies that a customer can activate to alertthe system that a target product the customer is interested in is out ofstock. For example, customer may push a button immediately proximate toshelf-space for a product to indicate the out-of-stock target productthe customer is interested in. The pressing of such a button could besensed by the system through an electronic communication between thebutton and system server, or the pressing of such a button could bedetermined by analysis of one or more images showing the customerpressing the button. In other embodiments, the customer may provide anindication of the desired target product that is out of stock through aninteraction with an extended reality appliance, such as by clicking onthe display of such extended reality appliance at location on a storeimage showing the empty shelf-space where the target product would be ifit were in stock. As used herein, the term “in-store product” means anyproduct that is ordinarily stocked by the store for purchase by in-storeshoppers. In some embodiments, one or more operating instructionsincluded in a non-transitory computer readable medium are executed by aprocessor of server 135 to make the second determination based on anautomatic analysis of image data acquired by one or more cameras locatedin the physical retail store, where in some embodiments the one or morecameras are mounted within the physical retail store, and in otherembodiments, the one or more cameras are included on a device (e.g., amobile device, phone, glasses, etc.) associated with the shopper. Thestore image may be analyzed to identify any unavailable in-store productby the presence of empty shelf space where the in-store product wouldnormally be available for retrieval by the shopper. In another exemplaryembodiment, the second determination may be made without analysis of animage, but instead rely upon other sensor means as described elsewhereherein.

In some embodiments, one or more operating instructions included in anon-transitory computer readable medium are executed by a processor ofserver 135 to make the second determination based on an automaticanalysis of an electronic inventory database associated with thephysical retail store. Such an automatic analysis may be based on acomparison of a customer shopping list with the electronic inventorydatabase to identify target products on the shopping list which areidentified as being out-of-stock in the store.

In an exemplary embodiment, once the second determination is made thatthe at least one product category action of the shopper is associatedwith a presently unavailable in-store product, the system may transmitto a device with a display a shopper notification that contains onlineplatform ordering information for the unavailable in-store product, or asubstitute product. Thus, in an example embodiment, one or moreoperating instructions included in a non-transitory computer readablemedium are executed by a processor of server 135 to, in response to thefirst and second determinations, provide a notification to the shopper,wherein the notification includes information regarding ordering thetarget product via an online platform.

Referring to FIG. 15C, the shopper notification may be provided to thecustomer via infrastructure of the physical retail store in a locationassociated with the category of products. For example, in some exemplaryembodiments, as illustrated in FIG. 15C, the shopper notification may betransmitted for display on a screen device 1240 that is near the shelfspace 1230 used for the unavailable in-store product. Alternatively,referring to FIGS. 15A and 15B, in an example embodiment a shoppernotification 1500 may be transmitted for display on screen 1300 of awearable extended reality device 1210 associated with the shopper 1505.Referring to FIGS. 16A and 16B, in another exemplary embodiment, shoppernotification 1500 may be transmitted for display on screen 1350 of ahand-held extended reality appliance 1210 associated with the shopper1505.

As illustrated in FIGS. 15A, 15B, 15C, 16A and 16B, the shoppernotification 1500 contains online platform ordering information for theunavailable in-store target product 1510. As illustrated, theunavailability of in-store product 1510 may be indicated with a graphicand/or text overlay of a contemporaneous display of an image of theempty shelf space where the target product would otherwise be expectedto be if it were available in-store.

Shopper notification 1500 may in some embodiments, as illustrated,include a price for the target product on the online platform. In someembodiments, the notification includes a price for an alternativeproduct on the online platform, wherein the alternative product isdifferent from the target product, but wherein the alternative productand the target product are both associated with a common manufacturerbrand.

In some embodiments, the price displayed in the shopper notification maybe a special affinity price that is determined by the system based on aknown affinity of the shopper for the particular unavailable in-storeproduct and the in-store price of the particular unavailable in-storeproduct. Such affinity pricing may be triggered, as illustrated in FIG.15C. For example, in some embodiments, the at least one product categoryaction may include picking up a particular product 1515 that is a directalternative to the target product and is associated with a differentmanufacturer brand than the target product. In such a situation, forexample in an effort to maintain shopper brand loyalty, the notificationmay include a price for the target product 1510, the price for thetarget product may be determined based on an affinity of the customer tothe target product and/or on a price of the particular product in thephysical retail store. In one example, the affinity of the customer tothe target product may be indicative of loyalness of the customer to thetarget product, and may be determined based on an analysis of pastpurchases of the customer. In one example, a lower price for the targetproduct may be selected when the customer has higher affinity to thetarget product (e.g., when the customer purchased the target productmany times in the past and/or when the customer never or rarely everpurchased alternative products to the target product). In one example, alower price for the target product may be selected when the price of theparticular product in the physical retail store is lower.

In some embodiments, perhaps because the unavailable in-store producthas been discontinued, and as illustrated in FIGS. 16A and 16B, theshopper notification 1500 may contain online platform orderinginformation for a substitute alternative product. The substitute productmay be a same-brand substitute product (e.g., the manufacturer hasintroduced a new version of an unavailable in-store product).Alternatively, the substitute product may be an off-brand substituteproduct (e.g., because the goods of the manufacturer for unavailablein-store product are no longer available in-store or through the onlineplatform).

In an example embodiment, where shopper notification 1500 is transmittedfor display to a shopper mobile device with an online platformconnection capability, shopper notification 1500 may, as illustrated,include a hyperlink (“Order Here”) to the target product on the onlineplatform.

FIG. 17 is a flowchart representing an exemplary method 1700, inaccordance with example embodiments of the present disclosure, forproviding an in-store shopper notification for the online ordering of anunavailable in-store product. For purposes of illustration, in thefollowing description, reference is made to certain components of system100 as deployed in the configuration depicted in FIG. 12 . For example,method 1700 may be performed by at least one processor or at least oneprocessing unit, such as processing unit 130 or processing device 202.It is to be understood that throughout the present disclosure, the term“processor” is used as a shorthand for “at least one processor.” Inother words, a processor may include one or more structures that performlogic operations whether such structures are collocated, connected, ordispersed. In some embodiments, a non-transitory computer readablemedium may contain instructions that when executed by a processor causethe processor to perform method 1700. Further, method 1700 is notnecessarily limited to the steps shown in FIG. 17 , and any steps orprocesses of the various embodiments described throughout the presentdisclosure may also be included in method 1700.

At step 1702, method 1700 includes a processor of server 135 executingone or more operating instructions located in a non-transitorycomputer-readable medium to make a first determination that a customerlocated in a physical retail store has engaged in at least one productcategory action associated with an interest in a category of productsassociated with the retail store. For example, by sensing where in astore aisle the shopper is located, and which shelf space is within thedirection of view for the shopper, and how long the shopper hasmaintained the direction of view, the system can make a firstdetermination that the in-store shopper is presently in the vicinity ofand/or focused on a particular product category (i.e., certain types ofproducts) and is taking at least one product category action.

At step 1704, a processor of server 135 executing one or more operatinginstructions located in a non-transitory computer-readable medium tomake a second determination that the at least one product categoryaction is associated with a target product, included in the category ofproducts, that is out of stock in the physical retail store. This seconddetermination may be based upon an identification of a target product ona customer shopping list stored in the system, prior shopper activity,an automatic analysis of image data acquired by one or more cameraslocated in the physical retail store, analysis of an inventory database,and/or sensor input concerning movement or placement of products onstore shelf by the shopper.

At step 1706, a processor of server 135 executing one or more operatinginstructions located in a non-transitory computer-readable medium to, inresponse to the first and second determinations, provide a notificationto the customer, wherein the notification includes information regardingordering the target product via an online platform. The shoppernotification may be provided to the customer via infrastructure of thephysical retail store, such as a shelf-mounted system display unit, orthrough an extended reality appliance associated with the in-storeshopper. The shopper notification may include online price informationas well as a hyperlink to place an order for the alternative productthrough the online platform.

Some shoppers may keep a virtual shopping list on their computer or ontheir mobile device (e.g., either stored directly on their computer, viaa Web site accessible by the computer, on their mobile device, on anapplication running on the mobile device, or via a Web site accessibleby the application running on the mobile device). The shoppingexperience may be enhanced if the shopper is able to know prior to goingto a retail store whether the store has the lowest total price for all(or a certain percentage of) the products on their shopping list and/orwhether the retail store has all (or a certain percentage of) theproducts on the shopping list. The shopper may prefer a particularretail store (e.g., a retail store close to the shopper's home), but theshopper may not mind traveling to a second retail store if they are ableto save money and/or find products that are not available at theirpreferred retail store. By combining the pricing and inventoryinformation collected by the methods and systems described herein withthe shopper's virtual shopping list (in some instances, includingshopper preferences for retail store and day and time of visiting theretail store), it may be possible to present retail store options to theshopper such that the shopper is able to save money on their totalpurchase and/or find all (or most of) the products on their virtualshopping list.

In some embodiments, a virtual shopping list of products is accessed bythe disclosed systems and methods, and the products in the shopping listmay be distributed over different retail stores. The distribution (orgrouping of products from the list by retail store and an expected timewindow of a user's arrival at each retail store) may be based on acombination of information, including price information and inventoryinformation for the products located at each retail store. For example,based on a determination of when a user is expected to frequent aparticular retail store (e.g., based on shopping habits, user input, aschedule or planned route, etc.), the system can analyze product pricingand inventory information for the expected shopping times,cross-correlate that information with the products on the shopping list,and provide recommendations for where and when to purchase products onthe shopping list.

In some embodiments, a customer of a shopping service may enter theshopping list, which may then be accessible by the user (e.g., a personshopping on behalf of the customer). In such embodiments, the systemsand methods described herein would perform in a similar manner.

Some embodiments provide a method for distributed shopping. In someembodiments, a non-transitory computer-readable medium includinginstructions that when executed by a processor may cause the processorto perform the method. In some embodiments, the method may be performedby a system for distributed shopping, the system including at least oneprocessing unit configured to perform the steps of the method. Forexample, as discussed above, the disclosed system may include one ormore servers 135, which may include one or more processing devices 202.Processing device 202 may be configured to execute one or moreinstructions stored in a non-transitory computer-readable storagemedium. As also discussed above, the non-transitory computer-readablemedium may include one or more of random access memory (RAM), read-onlymemory (ROM), volatile memory, nonvolatile memory, hard drives, CD ROMs,DVDs, flash drives, disks, any other optical data storage medium, anyphysical medium with patterns of holes, a RAM, a PROM, an EPROM, aFLASH-EPROM or any other flash memory, NVRAM, a cache, a register, anyother memory chip or cartridge, and networked versions of the same, etc.

In some embodiments, the method may include accessing a virtual shoppinglist associated with a user, the virtual shopping list identifying aplurality of products. As used herein, the terms “shopper” and “user”are interchangeable. In some embodiments, the virtual shopping list maybe included on a mobile device associated with the user (e.g., outputdevice 145D as shown in FIG. 1 ). In some embodiments, the user mayaccess his or her shopping list via a computer (e.g., a desktop computeror a laptop computer) and the virtual shopping list may be included in adatabase on a remotely located server. For example, the user may accessa Web site that stores a shopping list on the remotely located server.As another example, the user may access an application running on amobile device and the shopping list may be stored on the mobile deviceor on the remotely located server. The plurality of products can includeany products that the user wants to purchase or is contemplatingpurchasing. In some embodiments, a customer of a shopping service mayenter the shopping list (i.e., the virtual shopping list is associatedwith the customer of the shopping service), which may then be accessibleby the user (e.g., a person shopping on behalf of the customer).

In some embodiments, the method may include determining a firstindicator associated with a prospective visit by the user to a firstretail store and determining a second indicator associated with aprospective visit by the user to a second retail store. In someembodiments, the first retail store and the second retail store areassociated with a common retailer. For example, the first retail storeand the second retail store may be different stores for the sameretailer (e.g., a grocery store chain). In some embodiments, the firstretail store and the store retail store may be associated with differentretailers (e.g., different grocery store chains, a grocery store and abakery, a grocery store and a butcher shop, a grocery store and aclothing store, etc.). As used herein, the term “prospective visit” mayinclude a known or expected visit by the shopper to the retail store. Insome embodiments, a day and time for the prospective visit may be basedon the user's prior shopping history, which may be accessible by themethod if the user grants permission to access the shopping history. Insome embodiments, the day and time for the prospective visit may bebased on the user's prior travel history (e.g., daily commute time towork or if the user typically visits the retail store on the same day ofthe week, etc.), which may be accessible by the method if the usergrants permission to access the travel history.

In some embodiments, the first indicator identifies a first time windowassociated with the prospective visit by the user to the first retailstore and the second indicator identifies a second time windowassociated with the prospective visit by the user to the second retailstore. In some embodiments, the first time window of the first indicatorand the second time window of the second indicator are determined basedon input provided by the user. For example, the user may inputpreferences for the time windows via a Web site or into an applicationrunning on their mobile device. As another example, the user may grantpermission for the method to access a calendar, which may be stored onthe user's mobile device or on a remote server (e.g., a cloud-basedcalendar application). As another example, the user may grant permissionfor the method to access the user's driving history or travel history,which may be stored on a mobile device (e.g., accessing informationrelated to when the user travels to work).

In some embodiments, the first time window of the first indicator andthe second time window of the second indicator are determined based onanalysis of a prior shopping history associated with the user. Forexample, the user may grant permission for the method to access ashopping history associated with the first retail store and the secondretail store. In some embodiments, the user's shopping history may bestored in a centrally located database (e.g., database 140) accessibleby a server (e.g., server 135) that may be accessible to and used by themethod. In some embodiments, the user's shopping history for retailstores may be stored on the user's mobile device (e.g., output device145D) and the user may grant permission to the method to access theshopping history. In some embodiments, the user's shopping history forone or more retail stores may be stored separately on the user's mobiledevice (e.g., in an application specific to each retail store), and theuser may grant permission to the method to access each of thoseapplications to retrieve the shopping history.

In some embodiments, the method may include receiving pricinginformation for the plurality of products located at the first retailstore and at the second retail store. In some embodiments, the pricinginformation may be received via one of the methods described above(e.g., the product model used by server 135 or catalog data 244). Insome embodiments, the pricing information may be received directly fromthe first retail store and the second retail store.

In some embodiments, the first time window of the first indicator andthe second time window of the second indicator are determined based onanalysis of the received pricing information. In some embodiments, theprice of a product may change depending on the time of day. So that theuser may be able to purchase the product at a lowest possible price, thefirst time window and the second time window may be determined tocoincide with a time of day when the first retail store and the secondretail store offer the product at a low price. For example, with aperishable item (e.g., a freshly baked loaf of bread), a retail storemay discount the price of the freshly baked loaf of bread after acertain time of day (e.g., after 5:00 pm) such that the retail store maysell the freshly baked loaves of bread made during the day, to ensurethat the retail store may minimize or avoid discarding any unsoldfreshly baked loaves of bread. If the user has a freshly baked loaf ofbread on their shopping list, and with knowledge of the retail store'sprice reduction after 5:00 μm, the method may determine the first timewindow and the second time window to be after 5:00 pm to enable the userto buy the freshly baked loaf of bread at a discounted price.

In some embodiments, the received pricing information applies to thefirst and second time windows. In some embodiments, the price of aproduct may change depending on the time of day. To ensure that the userhas accurate price information for the time of day when the user islikely to visit the first retail store and/or the second retail store,the method may base the pricing information provided to the user basedon the user's preferred time of day for shopping. Continuing with theabove example of freshly baked loaf of bread being discounted after 5:00pm, if the user's preferred time window for shopping is around 8:00 am,then the method may provide the normal price of the freshly baked loafof bread. But if the user's preferred time window for shopping is after5:00 μm, then the method may provide the discounted price of the freshlybaked loaf of bread.

In some embodiments, the method may include receiving inventoryinformation for the plurality of products located at the first retailstore and at the second retail store. In some embodiments, the inventoryinformation may be received via one of the methods described above(e.g., inventory data 246 or method 1000). In some embodiments, theinventory information may be received directly from the first retailstore and the second retail store.

In some embodiments, the first time window of the first indicator andthe second time window of the second indicator are determined based onanalysis of the received inventory information. In some circumstances,the first retail store and/or the second retail store may sell out of aproduct that the user wants to buy or may have insufficient quantity onhand to fulfill the user's needs (e.g., the user wants four cans ofvegetable soup, but the retail store has either sold out or has lessthan four cans available). For example, the first time window and thesecond time window may be determined based on whether the first retailstore and/or the second retail store has a sufficient quantity of theparticular product on the user's shopping list. For example, the firsttime window and/or the second time window may be adjusted to coincidewith when the first retail store and/or the second retail store has atleast four cans of vegetable soup available for purchase.

In some embodiments, the received inventory information applies to thefirst and second time windows. To ensure that the user has accurateinventory information for the time of day when the user is likely tovisit the first retail store and/or the second retail store, the methodmay base the inventory information provided to the user on the user'spreferred time of day for shopping. Continuing with the above examplewhere the user wants to purchase four cans of vegetable soup, based onthe user's preferred time window for shopping, the method may update theinventory information close to the time window (e.g., in near real-time)such that the user has an accurate indication of whether the firstretail store or the second retail store has at least four cans ofvegetable soup.

In some embodiments, the method may include automatically generating anupdated virtual shopping list including an indication of a location forpurchasing each of the plurality of products, the indication of thelocation being associated with the first retail store or the secondretail store; wherein the updated virtual shopping list is generatedbased on a combination of the received pricing information and thereceived inventory information. The user's updated virtual shopping listmay indicate which retail stores the user should visit to purchasecertain products from the shopping list. For example, the updatedvirtual shopping list may include five products to be purchased from thefirst retail store and seven products to be purchased from the secondretail store. In some embodiments, the grouping of products to bepurchased at the first retail store and the second retail store may beadjusted so that the user's combined purchase price for all the productson the shopping list is minimized In some embodiments, the grouping ofproducts to be purchased at the first retail store and the second retailstore may be adjusted so that the user is able to purchase all theproducts on the shopping list (e.g., if a product is not available atthe first retail store, that product would be included in the list ofproducts to be purchased from the second retail store).

In some embodiments, the method may include providing the updatedvirtual shopping list to the user. In some embodiments, the updatedvirtual shopping list may be provided to the user via the Web site wherethe user initially entered their shopping list. In some embodiments, theupdated virtual shopping list is provided to the user via a mobiledevice associated with the user. For example, the updated virtualshopping list may be provided to the user via the application running onthe user's mobile device where the user initially entered a shoppinglist. In some embodiments, the updated virtual shopping list is providedto the user via a server-based shopping account associated with theuser. In some embodiments, the server-based shopping account may beaccessed by the user via a Web site or an application running on theuser's mobile device. In some embodiments where the customer of theshopping service enters the shopping list, the updated virtual shoppinglist may be provided to the user (e.g., the person shopping on behalf ofthe customer).

In some embodiments, the method may include receiving a forecast ofprospective freshness levels for a particular product, among theplurality of products, in the first retail store at different points intime in the future. As used herein, the term “freshness level for aparticular product” may refer to a particular product that has mostrecently been placed on display for sale. In other words, the morerecently the particular product has been placed on display for sale, thehigher the freshness level. For example, the shopper may prefer to buyproduce that was recently placed out for sale as opposed to produce thathas been sitting on the display for a period of time. In someembodiments, the forecast of the prospective freshness levels may bebased on an analysis of an image of the particular product in the firstretail store or the second retail store. The image analysis may beperformed by any suitable method (e.g., by image processing unit 130described above).

In some embodiments, a machine learning model may be trained usingtraining examples to determine prospective freshness levels from imagesand/or videos. An example of such training may include a sample imageand/or a sample video of a sample product, together with a labelindicating a prospective freshness level of the product at one or moreselected future time periods. In one example, the trained machinelearning model may be used to analyze the images captured in at leastone of the first retail store or the second retail store and determinethe prospective freshness levels for the particular product. In someexamples, a convolution of at least part of the images captured in atleast one of the first retail store or the second retail store may becalculated to obtain a result value of the calculated convolution. Inone example, the prospective freshness levels for the particular productmay be determined based on the result value of the calculatedconvolution. In some examples, image classification algorithms may beused to analyze the images captured in at least one of the first retailstore or the second retail store to classify products in at least one ofthe first retail store or the second retail store to differentprospective freshness levels.

In some embodiments, the method may include selecting the first timewindow based on the prospective freshness levels. For example, the firsttime window may be selected based on when fresh produce has been placedout for sale at the first retail store. In another example, the firsttime window may be selected to a time when the prospective freshnesslevels are above a selected threshold level. For example, theprospective freshness levels may be used to determine that a freshnesslevel of the particular product is predicted to drop below an acceptedlevel at a particular hour during the day, and as a result, a timewindow earlier than that particular hour may be selected as the firsttime window.

In some embodiments, the determined first indicator includes a firsttime window associated with the prospective visit by the user to thefirst retail store. For example, the first time window may be determinedbased on the user's history of prior visits to the first retail store.In some embodiments, the user may grant permission for the method toaccess the user's prior shopping history at the first retail store,e.g., by providing access to an application running on the user's mobiledevice or by providing access to the user's travel history stored ontheir mobile device.

In some embodiments, the determined second indicator includes a secondtime window associated with the prospective visit by the user to thesecond retail store. For example, the second time window may bedetermined based on the user's history of prior visits to the secondretail store in a similar manner as determined with the user's historyof prior visits to the first retail store.

In some embodiments, the first time window and the second time windoware automatically determined based on analysis of the virtual shoppinglist, the received pricing information, and the received inventoryinformation. For example, the products from the virtual shopping listmay be grouped by retail store location based on a cross-correlation ofprice and prospective inventory at the first retail store and the secondretail store. In this manner, the user's trip to the first retail storeand the second retail store may be optimized in that the user may beable to purchase at least some of the products on the virtual shoppinglist for a lowest possible total price while maximizing the likelihoodof being able to purchase all or a majority of the products (e.g., notencountering a situation where a retail store has no inventory orinsufficient inventory for one or more products) and the likelihood ofobtaining the freshest possible products.

In some embodiments, at least one of the first time window or the secondtime window is determined at least in part on a planned price reductionfor at least one of the plurality of products in either the first retailstore or the second retail store. For example, based on an analysis ofhistorical pricing information, it may be determined that the firstretail store and/or the second retail store implements a price reductionon certain products at certain days and times. Using one of the examplesabove, the first retail store may discount the price of a freshly bakedloaf of bread after a certain time of day (e.g., after 5:00 pm) suchthat the first retail store may sell most or all of the freshly bakedloaves of bread made during the day, to ensure that the first retailstore may minimize or avoid discarding unsold freshly baked loaves ofbread. If the user has a freshly baked loaf of bread on a shopping list,and with knowledge of the first retail store's price reduction after5:00 μm, the method may determine the first time window to be after 5:00pm to enable the user to buy the freshly baked loaf of bread at adiscounted price.

As another example, the second retail store may offer discounts oncertain products on a certain day of the week (e.g., on Friday). Withthis knowledge, the method may be able to match products on the user'svirtual shopping list with the products that will be discounted at thesecond retail store on Friday and may determine the second time windowto be on Friday when the discounted products are available. As anotherexample, if the second retail store offers different products at adiscount on a certain day of the week (e.g., different products arediscounted every Friday, not necessarily the same products each week),the method may be able to dynamically match products on the user'svirtual shopping list with the products that will be discounted at thesecond retail store on Friday and may determine the second time windowto be on Friday when the discounted products are available.

In some embodiments, at least one of the first time window or the secondtime window is determined at least in part on a planned increase ininventory for at least one of the plurality of products in either thefirst retail store or the second retail store. For example, the firstretail store may receive delivery of certain products on a certain dayof the week and those products may be placed on the shelves by a certaintime of the delivery day. As an example, first retail store receivesdelivery of cans of soup every Wednesday and the shelves are stockedwith the cans of soup by 11:00 am. If the user has one or more cans ofsoup on their virtual shopping list, the method may determine the firsttime window to be after 11:00 am on Wednesday so that the user may beable to purchase the cans of soup.

In some embodiments, the updated virtual shopping list is generatedbased on price comparison for a particular product, among the pluralityof products, in the first retail store and the second retail store. Forexample, the updated virtual shopping list may be divided into productsto be purchased at the first retail store and products to be purchasedat the second retail store. The grouping of the products between thefirst retail store and the second retail store may be adjusted tominimize the total cost of products on the updated virtual shoppinglist. For example, if the second retail store has three products on thevirtual shopping list at a lower price than the same three products atthe first retail store, then those three products may be placed on theupdated virtual shopping list associated with the second retail store.

In some embodiments, the user may set a threshold for the pricedifference between the two retail stores before the products are movedto the virtual shopping list for a different store. For example, theuser may set a price difference of at least $0.25 between the firstretail store and the second retail store as the threshold for movingproducts on the virtual shopping list. The user may find that it is notworth the extra time and travel costs to shop at two retail stores ifthe savings for traveling to the second retail store are insufficient.In some embodiments, the updated virtual shopping lists for the firstretail store and the second retail store may be dynamically adjustedbased on the price difference threshold and current travel times and/orcurrent fuel costs.

In some embodiments, the updated virtual shopping list is generatedbased on prospective inventory levels for a particular product, amongthe plurality of products, in the first retail store and the secondretail store. For example, the updated virtual shopping list may bedivided into products to be purchased at the first retail store andproducts to be purchased at the second retail store. The grouping of theproducts between the first retail store and the second retail store maybe adjusted to ensure that the user is able to purchase all or amajority of the products on the virtual shopping list, regardless ofwhether the products are purchased from the first retail store, thesecond retail store, or a combination of the first retail store and thesecond retail store. In some embodiments, the prospective inventorylevels may be based on planned deliveries to the first retail store andthe second retail store or a current rate of sales at the first retailstore and the second retail store. In some embodiments, the virtualshopping list may be dynamically updated based on the prospectiveinventory levels at the user's preferred time window. For example, ifthe user usually shops on Wednesday after 6:00 μm, the list of productsto be purchased from the first retail store and/or the second retailstore may be adjusted to reflect the prospective inventory levels at theretail stores on Wednesday after 6:00 μm.

In some embodiments, the updated virtual shopping list may be adjustedsuch that the user is not purchasing only one product at the firstretail store or the second retail store. For example, if a particularproduct is only available for purchase at the second retail store, themethod may move some additional products on the updated virtual shoppinglist from the first retail store to the second retail store. In someembodiments, the additional products may be moved on the updated virtualshopping list from the first retail store to the second retail store ifthe products have the same price at both retail stores. In someembodiments, the additional products to be moved on the updated virtualshopping list from the first retail store to the second retail store maybe based on user preferences. For example, the user may prefer topurchase certain products from the second retail store over purchasingthe same or similar products from the first retail store. In someembodiments, the additional products to be moved on the updated virtualshopping list from the first retail store to the second retail store maybe based on a combination of price, inventory, user preferences, and/orthe time windows. In some embodiments, the user may be prompted toconfirm that additional products on the updated virtual shopping listare to be moved from the first retail store to the second retail store.For example, the user may prefer to only shop at the first retail storeeven if there may be a cost savings by shopping for the one product atthe second retail store.

In some embodiments, the virtual shopping list may be updated in nearreal-time to reflect the current inventory at the first retail store andthe second retail store. For example, the virtual shopping list may beupdated before the user begins shopping at the first retail store. If,while the user is shopping, the first retail store is sold out of aparticular product, the method may update the virtual shopping list tomove the particular product from the shopping list for the first retailstore to the shopping list for the second retail store, so that the useris able to purchase the particular product. In some embodiments, themethod may provide an alert to the user (e.g., via the user's mobiledevice) that one or more products have been moved on the virtualshopping list from the shopping list for the first retail store to theshopping list for the second retail store. In some embodiments, the usermay override the movement of one or more products on the virtualshopping list from the shopping list for the first retail store to theshopping list for the second retail store. For example, if the user iscurrently shopping at the first retail store and has already placed aproduct in a shopping basket (and the inventory information for thefirst retail store is updated in near real-time to indicate that thefirst retail store is now sold out of the product), the user would notwant that product moved to the shopping list for the second retailstore. In such circumstances, the user may override moving that productto the shopping list for the second retail store.

In some embodiments, the updated virtual shopping list is generatedbased on a prospective freshness level for a particular product, amongthe plurality of products, in the first retail store and the secondretail store. For example, the shopper may prefer to buy produce thatwas recently placed out for sale as opposed to produce that has beensitting on the display for a period of time. In some embodiments, theparticular product (e.g., produce) may be moved on the updated virtualshopping list from the first retail store to the second retail store ifthe second retail store has more recently placed fresh produce out forsale. In some embodiments, the virtual shopping list may be updateddynamically close to the first time window or the second time windowsuch that the virtual shopping list may be updated in near real-time topermit the user to buy the particular product at a highest freshnesslevel (i.e., most recently placed on display for sale).

In some embodiments, the updated virtual shopping listed may begenerated based on whether one or more products are perishable. Forexample, if the updated virtual shopping list indicates that the user isshopping at both the first retail store and the second retail store andthe shopping list includes ice cream, the virtual shopping list may beupdated to place the ice cream on the shopping list for the retail storewhere the user will be stopping last. For example, if the user is firstshopping at the first retail store and then shopping at the secondretail store, the virtual shopping list may be updated to place the icecream on the shopping list for the second retail store so that the icecream won't melt before the user is able to return home.

In some embodiments, the inventory information for the plurality ofproducts is determined based on analysis of images captured in at leastone of the first retail store or the second retail store. The imageanalysis may be performed by any suitable method (e.g., by imageprocessing unit 130 described above). The image analysis may beperformed periodically (e.g., as described above in connection with anyone or more of FIG. 4A, 4C, 6A, 7A, or 7B). In some embodiments, theinventory information determined in this manner may be used to generatethe updated virtual shopping list. For example, if the inventoryinformation is determined in near real-time, then the virtual shoppinglist may be updated in near real-time. In this way, the user may havethe most current information available so that the user may purchase allthe products on the virtual shopping list.

In some embodiments, a machine learning model may be trained usingtraining examples to determine inventory information from images and/orvideos. An example of such training may include a sample image and/or asample video captured from a sample retail store, together with a labelindicating inventory information corresponding to the sample retailstore. In one example, the trained machine learning model may be used toanalyze the images captured in at least one of the first retail store orthe second retail store and determine the inventory information for theplurality of products. In some examples, a convolution of at least partof the images captured in at least one of the first retail store or thesecond retail store may be calculated to obtain a result value of thecalculated convolution. In one example, the inventory information forthe plurality of products may be determined based on the result value ofthe calculated convolution. In another example, in response to theresult value of the calculated convolution being a first value, a firstinventory information for the plurality of products may be determined,and in response to the result value of the calculated convolution beinga second value, a second inventory information for the plurality ofproducts may be determined, the second inventory information may differfrom the first inventory information. In some examples, productrecognition algorithms may be used to analyze the images captured in atleast one of the first retail store or the second retail store torecognize products in at least one of the first retail store or thesecond retail store, and the inventory information for the plurality ofproducts may be determined based on the recognized products.

In some embodiments, the pricing information for the plurality ofproducts is determined based on analysis of images captured in at leastone of the first retail store or the second retail store. The imageanalysis may be performed by any suitable method (e.g., by imageprocessing unit 130 described above). The image analysis may beperformed periodically (e.g., as described above in connection with anyone or more of FIG. 4A, 4C, 6A, 7A, or 7B). In some embodiments, thepricing information determined in this manner may be used to generatethe updated virtual shopping list. For example, if the pricinginformation is determined in near real-time, then the virtual shoppinglist may be updated in near real-time. In this way, the user may havethe most current information available so that the user may purchase allthe products on the virtual shopping list for a lowest possible totalprice.

In some embodiments, a machine learning model may be trained usingtraining examples to determine pricing information from images and/orvideos. An example of such training may include a sample image and/or asample video captured from a sample retail store, together with a labelindicating pricing information corresponding to the sample retail store.In one example, the trained machine learning model may be used toanalyze the images captured in at least one of the first retail store orthe second retail store and determine the pricing information for theplurality of products. In some examples, a convolution of at least partof the images captured in at least one of the first retail store or thesecond retail store may be calculated to obtain a result value of thecalculated convolution. In one example, the pricing information for theplurality of products may be determined based on the result value of thecalculated convolution. In another example, in response to the resultvalue of the calculated convolution being a first value, a first pricinginformation for the plurality of products may be determined, and inresponse to the result value of the calculated convolution being asecond value, a second pricing information for the plurality of productsmay be determined, the second pricing information may differ from thefirst pricing information. In some examples, optical characterrecognition (OCR) algorithms may be used to analyze the images capturedin at least one of the first retail store or the second retail store toidentify prices in at least one of the first retail store or the secondretail store, and the pricing information for the plurality of productsmay be determined based on the identified prices.

In some embodiments, the updated virtual shopping list is provided tothe user together with a suggestion to purchase at least one of theplurality of products online. For example, if neither the first retailstore nor the second retail store has a particular product in inventory,the method may suggest that the user purchase the particular product viaan online store. In some embodiments, the online store is an onlinebranch of either the first retail store or the second retail store. Insome embodiments, the online store is not affiliated with either thefirst retail store or the second retail store.

In some embodiments, the virtual shopping list is updated based on thepricing information and a distance between the first retail store andthe second retail store. For example, if the cost of driving from theuser's home to the first retail store, then to the second retail store,and then to the user's home or other destination is greater than anycost savings by buying certain products at the second retail store, thevirtual shopping list may indicate that all products should be purchasedat the first retail store.

In some embodiments, an update to the inventory information for theplurality of products corresponding to an availability of a particularproduct at the second retail store may be received while the user isshopping at the first retail store. For example, the update to theinventory information may be received from a database. In anotherexample, the update to the inventory information may be based on ananalysis of an image captured in the second retail store while the useris shopping at the first retail store. The image analysis may includeany of the techniques described above. In response to the receivedupdate to the inventory information, a notification may be provided tothe user while the user is shopping at the first retail store. Forexample, the update may indicate an unexpected availability of theparticular product at the second retail store, and as a result thenotification provided to the user may include a recommendation topurchase the particular product at the second retail store even when theupdated virtual shopping list originally indicated the location forpurchasing the particular product as the first retail store. In oneexample, the notification may be provided to the user after the userpicked the particular product at the first retail store, and thenotification may include a recommendation to return the particularproduct. In an alternative example, the update may indicate anunexpected unavailability of the particular product at the second retailstore, and as a result, the notification provided to the user mayinclude a recommendation to purchase the particular product at the firstretail store even when the updated virtual shopping list originallyindicated the location for purchasing the particular product as thesecond retail store.

In some embodiments, an update to the pricing information for theplurality of products corresponding to an availability of a particularproduct at the second retail store may be received while the user isshopping at the first retail store. For example, the update to thepricing information may be received from a database. In another example,the update to the pricing information may be based on an analysis of animage captured in the second retail store while the user is shopping atthe first retail store. The image analysis may include any of thetechniques described above. In response to the received update to thepricing information, a notification may be provided to the user whilethe user is shopping at the first retail store. For example, the updatemay indicate a lower than expected price of the particular product atthe second retail store, and as a result the notification provided tothe user may include a recommendation to purchase the particular productat the second retail store even when the updated virtual shopping listoriginally indicated the location for purchasing the particular productas the first retail store. In one example, the notification may beprovided to the user after the user picked the particular product at thefirst retail store, and the notification may include a recommendation toreturn the particular product. In an alternative example, the update mayindicate a higher than expected price of the particular product at thesecond retail store, and as a result, the notification provided to theuser may include a recommendation to purchase the particular product atthe first retail store even when the updated virtual shopping listoriginally indicated the location for purchasing the particular productas the second retail store.

FIG. 18 is an example map 1800 showing possible routes for a user totake to visit the first retail store and the second retail store. Asshown in FIG. 18 , the method may determine routes from the user's home1802 to first retail store (Store 1) 1804 (e.g., route 1810), from firstretail store 1804 to second retail store (Store 2) 1806 (e.g., route1812), and from second retail store 1806 back to the user's home 1802(e.g., route 1814). It is noted that routes 1810, 1812, and 1814 areexemplary and that other routes may be determined. The method may basethe cost of the routes on various factors, for example, a cost of fueland/or an amount of time it takes to drive the routes. For example, ifthe cost of driving routes 1812 and 1814 is more expensive than thepotential cost savings by purchasing certain products at second retailstore 1806, then the virtual shopping list may indicate that the usershould purchase all or a majority of products on the shopping list atfirst retail store 1804. As another example, if the cost of drivingroutes 1812 and 1814 is less than the potential cost savings bypurchasing certain products at second retail store 1806, then thevirtual shopping list may indicate that the user should purchase someproducts at first retail store 1804 and other products at second retailstore 1806.

In some embodiments, if the user has one or more perishable items (e.g.,ice cream) on a shopping list, the method may update the virtualshopping list based on respective lengths of the routes 1810, 1812, and1814. For example, if the pricing information indicates that the priceof ice cream is cheaper at first retail store 1804 and if route 1814 islonger than route 1810, then the updated virtual shopping list mayindicate that the user should travel to second retail store 1806 firstand then travel to first retail store 1804 so that the user may returnhome as quickly as possible after purchasing the ice cream. It is notedthat the designations “first retail store” and “second retail store” arenot fixed and may be based on which retail store the user should visitfirst and which retail store the user should visit second. For example,a retail store located closest to the user's home 1802 may be designatedas the first retail store if the user begins their shopping trip fromhome. But the retail store closest to the user's home 1802 may bedesignated as the second retail store if the user begins their shoppingtrip from another location (e.g., from work), such that another retailstore (e.g., Store 2 1806) is closer to the user's starting point.

While FIG. 18 illustrates different routes using the user's home 1802 asthe starting point, in some embodiments, different starting points maybe used and the method will perform in a similar manner. For example,the starting point may be where the user works. As another example, thestarting point may be a current geographical location of the user whenthe method is accessed or at a particular time of day. The user mayprovide an indication to the method that the user would like to begintheir shopping trip (e.g., by clicking on a “shop now” button or similaruser interface element). In some embodiments, the starting point may beautomatically determined as the user's current geographical locationbased on the user pre-setting the time windows. For example, the usermay set the first time window and/or the second time window to begin at9:00 am on Sunday morning. At 9:00 am on Sunday morning, the method mayautomatically begin and may determine the user's geographic location atthat time, receive the pricing information and the inventoryinformation, and may generate the updated virtual shopping list to beprovided to the user.

In some embodiments, the user may specify a different end point (i.e.,the user does not start and end in the same location). For example, theuser may do their shopping on their way home from work, such that thestarting point is the user's place of work and the ending point is theuser's home. In some embodiments, the method may prompt the user toconfirm the ending point before generating the updated virtual shoppinglist. For example, a user interface element may appear on the user'smobile device and present one or more choices for the ending point tothe user. Once the user selects the ending point, the method receivesthe pricing information and the inventory information, and generates theupdated virtual shopping list.

FIG. 19A is a schematic illustration of a shopping list. For example,the user may enter their shopping list 1902, identifying a plurality ofproducts, in an application 1900 running on the user's mobile device(e.g., output device 145D). As another example, the user may entershopping list 1902 on a computer or via a server-based shopping accountassociated with the user. As another example, a customer of a shoppingservice may enter the shopping list, which may then be accessible by theuser (e.g., a person shopping on behalf of the customer).

FIG. 19B is a schematic illustration of an exemplary output of a methodfor distributed shopping. For example, an output screen 1920 may beprovided to the user including a list of products 1922 to be purchasedat the first retail store, a list of products 1924 to be purchased atthe second retail store, and a message 1926 indicating an amount ofsavings to the user by shopping for the products as assigned by themethod. In some embodiments, message 1926 may provide additionalinformation or different information to the user. For example, message1926 may indicate when the prices for the products were determined orwhen the inventory for the products was determined. In some embodiments,message 1926 may be highlighted with certain user interface elements tomore readily distinguish message 1926 from the rest of output screen1920. For example, message 1926 may be surrounded by a border, in adifferent font, in a different type size, in a different color, or withother graphic embellishments that may be defined by user interfaceguidelines associated with the device where the output screen 1920 isdisplayed (e.g., output device 145D).

FIG. 19C is a schematic illustration of a second exemplary output of amethod for distributed shopping. For example, an output screen 1940 maybe provided to the user including a list of products 1942 to bepurchased at the first retail store, a first user interface elements1944 that may be used to condense list 1942, a condensed list ofproducts 1946 to be purchased at the second retail store, a second userinterface element 1948 that may be used to expand list 1946, and amessage 1950 indicating an amount of savings to the user by shopping forthe products as assigned by the method. In some embodiments, message1950 may provide additional information or different information to theuser, similar to message 1926 described above. In some embodiments,first user interface element 1944 and second user interface element 1948may be any type of user interface element configured to expand orcollapse a list of items, such as lists 1942, 1946. While first userinterface element 1944 and second user interface element 1948 are shownin FIG. 19C as arrows, any type of user interface element may be used,consistent with user interface guidelines associated with the devicewhere the output screen 1940 is displayed (e.g., output device 145D).

It is noted that output screen 1940 contains similar information asoutput screen 1920 but is displayed in a different format. The outputformat of lists 1942, 1946 may be based on output parameters orconstraints of output device 145D or may be determined based on userpreferences (e.g., the user may prefer to see a side-by-side comparisonof the lists 1922, 1924 as shown in FIG. 19B or may prefer to see onelist 1942, 1946 at a time as shown in FIG. 19C). Other possible formatsfor displaying lists 1942, 1946 to the user are possible and are notlimited to the examples shown herein. The format used for displayinglists 1942, 1946 does not change the operation of the method describedherein.

FIG. 19D is a schematic illustration of a third exemplary output of amethod for distributed shopping. For example, an output screen 1960 maybe provided to the user including a list of products 1962 to bepurchased at the first retail store. As shown in FIG. 19D, the items onthe virtual shopping list are to be purchased from the first retailstore. In some embodiments, list of products 1962 may not be fullydisplayed (i.e., condensed) and may include a user interface element1964 to enable the user to see the rest of the products on list ofproducts 1962. An indicator 1966 of the second retail store may beprovided for completeness of output screen 1960. In some embodiments,indicator 1966 may be surrounded by a border, in a different font, in adifferent type size, in a different color, or with other graphicembellishments that may be defined by user interface guidelinesassociated with the device where the output screen 1960 is displayed(e.g., output device 145D). In some embodiments, indicator 1966 isoptional and may not be shown on output screen 1960. In suchembodiments, not showing indicator 1966 may reduce visual clutter onoutput screen 1960 and may reduce the possibility of user confusion.

A message 1968 may be provided in output screen 1960, indicating anamount of savings to the user by shopping for the products as assignedby the method. In some embodiments, message 1968 may provide additionalinformation or different information to the user, similar to message1926 described above. In some embodiments, message 1968 is optional andmay not be shown on output screen 1960.

FIG. 20 is a flowchart of an exemplary method 2000 for distributedshopping. Method 2000 may be performed by at least one processor or atleast one processing unit, such as processing unit 130, processingdevice 202. It is to be understood that throughout the presentdisclosure, the term “processor” is used as a shorthand for “at leastone processor.” In other words, a processor may include one or morestructures that perform logic operations whether such structures arecollocated, connected, or dispersed. In some embodiments, anon-transitory computer readable medium may contain instructions thatwhen executed by a processor cause the processor to perform method 2000.Further, method 2000 is not necessarily limited to the steps shown inFIG. 20 , and any steps or processes of the various embodimentsdescribed throughout the present disclosure may also be included inprocess 2000.

In step 2002, method 2000 accesses a user's virtual shopping list. Asdescribed above, the virtual shopping list is associated with the user,for example via a Web site or the user's mobile device. The virtualshopping list identifies a plurality of products that the user wishes topurchase. In some embodiments, a customer of a shopping service mayenter the shopping list, which may then be accessible by the user (e.g.,a person shopping on behalf of the customer).

In step 2004, method 2000 determines one or more indicators. Forexample, a first indicator may be associated with a prospective visit bythe user to a first retail store and a second indicator may beassociated with a prospective visit by the user to a second retailstore. As noted above, the first indicator may identify a first timewindow when the user may visit the first retail store and the secondindicator may identify a second time window when the user may visit thesecond retail store.

In step 2006, method 2000 receives pricing information for the pluralityof products. The pricing information may be received for productslocated at the first retail store, products located at the second retailstore, or products located at both the first retail store and the secondretail store such that a price comparison may be performed.

In step 2008, method 2000 receives inventory information for theplurality of products. The inventory information may be received forproducts located at the first retail store, products located at thesecond retail store, or products located at both the first retail storeand the second retail store such that an inventory comparison may beperformed.

In step 2010, method 200 automatically generates an updated virtualshopping list. The updated virtual shopping list may be generated usingone or more of the first indicator, the second indicator, the pricinginformation, or the inventory information. In some embodiments, thecombination of the first time window, the second time window, thepricing information, and the inventory information may be used todetermine a first subset of the plurality of products to be purchased atthe first retail store and a second subset of the plurality of productsto be purchased at the second retail store (i.e., distribute theshopping between the first retail store and the second retail store) tominimize the total purchase price for all or a majority of the pluralityof products. As another example, the combination may be used todistribute the shopping between the first retail store and the secondretail store to ensure that the user is able to purchase all theplurality of products.

In step 2012, method 2000 provides the updated virtual shopping list tothe user. As described above, the updated virtual shopping list may beprovided to the user via a computer (and a server-based shopping accountassociated with the user) or via a mobile device (e.g., via anapplication running on the user's mobile device).

Practical frictionless shopping systems are not 100% accurate. Forexample, a system may fail to identify the product type of a productselected or returned by a customer. In such situations, it may be usefulto query the customer about the product type, rather than imposing anon-frictionless solution. However, customers may vary in terms oftrustworthiness. In some embodiments, a test event may be used toestimate a customer's trustworthiness, and based on the outcome of thetest event (i.e., whether the customer is determined to be trustworthyor untrustworthy), an appropriate action may be selected in response toa future, ambiguous product selection event. For example, where a testevent includes querying a customer about a recognized product selectionevent (e.g., regarding price, quantity, product type, etc.), and thecustomer responds accurately, that customer may be provided with africtionless shopping option (e.g., a query and trusted response by thecustomer) in response to future ambiguous selection events. On the otherhand, if a customer does not provide an accurate response to the testevent query, a less frictionless approach may be employed in response tofuture ambiguous selection events.

Some embodiments provide a frictionless shopping method based onestimated shopper integrity. In some embodiments, a non-transitorycomputer-readable medium including instructions that when executed by aprocessor may cause the processor to perform the method. In someembodiments, the method may be performed by a system for frictionlessshopping, the system including at least one processor configured toperform the steps of the method. For example, as discussed above, thedisclosed system may include one or more servers 135, which may includeone or more processing devices 202. Processing device 202 may beconfigured to execute one or more instructions stored in anon-transitory computer-readable storage medium. As also discussedabove, the non-transitory computer-readable medium may include one ormore of random access memory (RAM), read-only memory (ROM), volatilememory, nonvolatile memory, hard drives, CD ROMs, DVDs, flash drives,disks, any other optical data storage medium, any physical medium withpatterns of holes, a RAM, a PROM, an EPROM, a FLASH-EPROM or any otherflash memory, NVRAM, a cache, a register, any other memory chip orcartridge, and networked versions of the same, etc.

In some embodiments, the method may include receiving, from at least afirst of a plurality of sensors associated with a retail store, firstsensed data indicating a selection of a first product by a customer. Theplurality of sensors may include any type of sensors described herein,such as an image sensor (e.g., as a part of an image capture device suchas image capture device 506), a motion sensor, a sound sensor, aproximity sensor, a temperature sensor, a biometric sensor, a pressuresensor, a weight sensor, a light sensor, a resistive sensor, acapacitive sensor, an inductive sensor, a vacuum pressure sensor, a highpressure sensor, a conductive pressure sensor, an infrared sensor, aphoto resistor sensor, a photo-transistor sensor, a photo-diode sensor,an ultrasonic sensor, or the like. The plurality of sensors may beplaced in multiple locations throughout the retail store (e.g., on eachretail shelf or product display) and/or be placed in a shopping cartused by a customer (e.g., the sensors may be physically connected to theshopping cart).

In some embodiments, the plurality of sensors associated with the retailstore may include a camera, and wherein the first sensed data mayinclude image data. For example, the camera may include image capturedevice 506 described herein. In some embodiments, the plurality ofsensors may include multiple cameras and the cameras may be placed invarious locations in the retail store, such as along a retail shelf asshown in FIGS. 4A, 4C, 6A, and 6B). As another example, the camera maybe physically connected to a shopping cart used by the customer. In oneexample, the image data included in the first sensed data may include atleast one of: an image of the customer picking one or more items of thefirst product, an image of the customer placing one or more items of thefirst product in a container (such as a shopping cart, a bag, a basket,and so forth), or an image of the customer holding one or more items ofthe first product. In another example, the image data included in thefirst sensed data may include an image of one or more items of the firstproduct in a container (such as a shopping cart, a bag, a basket, and soforth). In yet another example, the image data included in the firstsensed data may include an image of one or more items of the firstproduct on a retail shelf and/or an image of an empty space on a retailshelf where one or more items are supposed to be placed and/or werepreviously placed on a retail shelf. In some examples, a machinelearning model may be trained using training examples to determineproperties associated with products from images and/or videos. Anexample of such training may include sample image data associated with asample product, together with a label indicative of a propertyassociated with the sample product. In one example, the trained machinelearning model may be used to analyze the image data included in thefirst sensed data and determine the first property associated with thefirst product. In some examples, a convolution of at least part of theimage data included in the first sensed data may be calculated to obtaina result value of the calculated convolution. In one example, the resultvalue of the calculated convolution may be used to determine the firstproperty associated with the first product. In one example, in responseto the result value of the calculated convolution being a first value,the first property associated with the first product may be determinedto have one value, in response to the result value of the calculatedconvolution being a second value, the first property associated with thefirst product may be determined to have a different value.

In some embodiments, the plurality of sensors associated with the retailstore may include a pressure sensor. In some embodiments, there may bemultiple pressure sensors. For example, the pressure sensors may includedetection elements 801A, 801B as shown in FIG. 8A. In some embodiments,the first sensed data may include a change in pressure associated withat least one region of a product display. As described in connectionwith FIG. 8A, the pressure sensor may operate based on physical contact(or lack thereof) with products placed on the at least one region of theproduct display. For example, if the product is removed from the productdisplay by a shopper, the pressure sensor can detect the change inpressure.

In some embodiments, the plurality of sensors may include at least onebarcode scanner. In some embodiments, there may be multiple barcodescanners and the scanners may be placed in various locations in theretail store, such as along a retail shelf. In some embodiments, thebarcode scanner may be implemented by an image capture device (e.g.,image capture device 506). In some embodiments, the barcode scanner maybe configured to scan a variety of barcode types, such asone-dimensional barcodes (e.g., Code 39, Code 128, Universal ProductCode (UPC), International Article Number (EAN), and the like) ortwo-dimensional barcodes (e.g., PDF417, Data Matrix, Quick Response (QR)Codes, and the like).

In some embodiments, the plurality of sensors associated with the retailstore may include a sensor configured with at least one of a retailshelf or a shopping cart of the retail store. As discussed above, thesensors may be placed in various location in the retail store, such asalong a retail shelf or a product display. In some embodiments, thesensors may be placed in a shopping cart (e.g., the sensors may bephysically connected to the shopping cart). In some embodiments, one ormore pressure sensors may be located at a bottom portion of the cartbasket, such that the pressure sensors can detect a total weight of allthe products placed in the shopping cart. For example, the pressuresensor may be configured as a single pressure pad or as a plurality ofindividual pressure sensors arranged in a grid or other pattern.

In some embodiments, one or more image capture devices may be connectedto various locations of the shopping cart such that the image capturedevices may take pictures or scan barcodes of all the products placed inthe shopping cart. For example, if the image capture device isconfigured to scan a barcode, by placing multiple image capture devicesaround a perimeter of the shopping cart (and facing toward an interiorof the shopping cart), the barcode may be scanned regardless of theorientation of the product when placed in the shopping cart. In otherwords, the customer does not have to manually scan the barcode of eachproduct placed in the shopping cart by positioning the barcode of theselected product to face a particular image capture device.

In some embodiments, the method may include determining, from the firstsensed data, a first property associated with the first product. In someembodiments, the property associated with the first product may includeone or more characteristics of the product, such as one or more visuallydiscernable features attributed to the product or one or more featuresthat may be detected by one or more of the plurality of sensors (e.g.,size of the product, shape of the product, information on a labelassociated with the product, weight of the product, etc.). In someembodiments, multiple properties associated with the first product maybe determined from the first sensed data. For example, if the firstsensed data includes a barcode of the first product, it may be possibleto determine multiple properties associated with the first product, suchas product type, brand, weight, and price. In some examples, a machinelearning model may be trained using training examples to determineproperties associated with products from sensed data. An example of suchtraining may include sample sensed data associated with a sampleproduct, together with a label indicative of a property associated withthe sample product. In one example, the trained machine learning modelmay be used to analyze the first sensed data and determine the firstproperty associated with the first product.

In some embodiments, the first property may be indicative of a producttype associated with the first product. In some embodiments, the producttype associated with the first product may include a broad category ofthe product type, e.g., a bottle of soda, a loaf of bread, a notepad, orthe like.

In some embodiments, the first property may be indicative of a quantityassociated with the first product. For example, the first property mayindicate that the customer selected a six-pack of soda cans or a packageof three cans of vegetable soup. In one example, the first property mayindicate whether the customer selected a first number of units or asecond number of units (for example, five or six, one or two, etc.) ofthe first product. In one example, the first property may indicate aweight of the first product. For example, the first product may be acustom-made product that may have different weights, and the firstproperty may indicate a weight of the first product. In one example, thefirst property may indicate a volume of the first product. For example,the first product may be a custom-made product that may have differentvolumes, and the first property may indicate a volume of the firstproduct.

In some embodiments, the first property may be indicative of a priceassociated with the first product. For example, the first property maybe the “shelf price” of the first product, where the “shelf price” isthe price of the first product as posted on the retail shelf. In someembodiments, the first property may be a discounted price of the firstproduct if, for example, the first product is currently being sold at adiscount by the retail store. In some embodiments, the first propertymay be a discounted price of the first product if, for example, thecustomer is identified as a “frequent shopper” at the retail store(e.g., via a retail store loyalty program) and the retail store providesa discounted price to customers identified as “frequent shoppers.”

In some embodiments, the method may include issuing a first query to thecustomer with respect to the first property associated with the firstproduct and, in response, receiving first information from the customer.The first query may be a “test event” to estimate the customer'strustworthiness based on the received first information. The customer'strustworthiness may be used to determine an action to be taken in afuture product selection event, as will be described in further detailbelow.

In some embodiments where multiple properties associated with the firstproduct may be determined from the first sensed data (e.g., based on ascanned barcode of the first product), the first query may be based onany one of the properties associated with the first product. In someembodiments, the property of the first product to be included in thefirst query may be predetermined. For example, the method may be setsuch that the first query relates to a price of the first product. Insome embodiments, the property of the first product to be included inthe first query may be determined at random from the multipleproperties.

In some embodiments, the first information received from the customermay include a selection of at least one of a plurality of optionspresented to the customer. For example, the customer may be prompted toanswer a question about the first product. In some embodiments, thefirst information may be received from a mobile device associated withthe customer. For example, the customer may receive a question about thefirst product on the mobile device and may be presented with a pluralityof options to select from. In some embodiments, the first informationmay be received from a communication device physically connected to ashopping cart Similar to the above example, the customer may receive aquestion about the first product on the communication device connectedto the shopping cart and may be presented with a plurality of options toselect from.

In some embodiments, the first information received from the customermay indicate at least one of a price, a selected quantity, or a producttype for the first product. For example, the first query may prompt thecustomer to confirm the “shelf price” of the first product. As anotherexample, the first query may prompt the customer to confirm whether thefirst product was a package of three cans of vegetable soup. As anotherexample, the first query may prompt the customer to confirm whether thefirst product was a box of cookies or a box of pasta.

In some embodiments, the first query may be presented as amultiple-choice question to enable easy customer response. For example,if the first query prompts the customer to confirm the price of thefirst product, the first query may present two or more choices for theprice of the first product. In some embodiments, the number of choicesmay be limited based on the screen size of the device where the firstquery is displayed. For example, if the first query is displayed on thecustomer's mobile device, the number of choices may be limited to two orthree. As another example, if the first query is displayed on thecommunication device connected to the shopping cart, the number ofchoices may be limited to four. In some embodiments, the first query mayprompt the customer to enter a number, for example, if the first queryis about the quantity of the first product selected by the customer. Thecustomer may then be prompted to enter a number on a numeric keypaddisplayed adjacent to the first query. It is noted that the format ofthe first query and how the customer enters the first information inresponse to the first query as described herein may be presented in anyformat capable of being used by the customer's mobile device or thecommunication device connected to the shopping cart. For example, theformat for the display may be based on graphical user interface elementsdefined by the display device or by an operating system running on thedisplay device. For example, the graphical user interface elements mayinclude buttons, radio buttons, or a text field.

In some embodiments, the method may include comparing the firstinformation received from the customer with the first propertyassociated with the first product. The comparing may be performed by anyknown comparison means. For example, the comparing may be performed by aremote server that receives the first information. As another example,the comparing may be performed by the customer's mobile device if thecustomer received the first query on the mobile device. As anotherexample, the comparing may be performed by the device connected to theshopping cart if the customer received the first query on the device. Asanother example, the comparing may be performed by a server located inthe retail store.

In some embodiments, the method may include determining, based on thecomparison between the first information received from the customer withthe first property associated with the first product, an integrityindicator associated with the customer. The integrity indicator may beany type of value that may be used to track a characteristic of thecustomer. In some embodiments, the integrity indicator may include avalue, an index, a measure, a flag, or the like. For example, theintegrity indicator may be a binary flag (e.g., a single bit that may beone of two possible values) such that the flag signifies whether thecustomer provided an accurate response or an inaccurate response. Asanother example, the integrity indicator may be a value within a rangeof possible values, e.g., based on how accurate or inaccurate the firstinformation is to the correct answer. As another example, integrityindicator may be a value that indicates a percentage difference betweenthe first information and the correct answer. For example, if the firstquery requires the customer to respond with a price of the firstproduct, the integrity indictor may be an absolute difference betweenthe first information (i.e., the price provided by the customer) and theactual price of the first product or may be a percentage differencebetween the price provided by the customer and the actual price of thefirst product.

In some embodiments, how the integrity indicator is determined may bebased on a characteristic of the first product. For example, if thefirst sensed data (e.g., a barcode scan) indicates that the price of thefirst product is greater than a threshold amount (e.g., greater than$10), then the first query may include asking the customer to confirmthe price of the first product and if the first information (e.g., theprice provided by the customer) is within 2% of the actual price of thefirst product, the integrity indicator may indicate that the customer istrustworthy. As another example, if the price of the first product isless than a threshold amount (e.g., less than $2), the first query mayinclude asking the customer to confirm the price of the first productand if the first information (e.g., the price provided by the customer)is within 5% of the actual price of the first product, the integrityindicator may indicate that the customer is trustworthy. As anotherexample, if the price of the first product is greater than a thresholdamount (e.g., greater than $10), then the first query may include askingthe customer about a different characteristic of the first product(e.g., about any property other than the price of the first product).The above examples may be expanded upon to include many variations basedon thresholds relating to certain product characteristics withoutchanging the overall operation of the method. For example, there may bea price threshold, a weight threshold, or a quantity threshold and thefirst query may relate to the same threshold (e.g., if using a pricethreshold, the first query may relate to the price of the first product)or the first query may relate to a different product characteristic.

In some embodiments, the integrity indicator may be stored temporarily,e.g., for a current shopping “session” while the customer is physicallylocated in the retail store and the integrity indicator may be reset toa default value when the customer leaves the retail store. In someembodiments, the integrity indicator may be stored for a longer periodof time, e.g., the integrity indicator may be a cumulative indicatorassociated with the customer over time (e.g., hours, days, weeks,months, years). For example, if the integrity indicator indicates thatthe customer provides an accurate response based on a threshold numberof queries, the method may randomly determine whether to issue the firstquery to the customer. In some embodiments, the threshold number ofqueries may be an absolute value, e.g., the customer has provided atleast 50 accurate responses. In some embodiments, the threshold numberof queries may be a relative value, e.g., the customer has providedaccurate responses to at least 95% of all queries. In some embodiments,the threshold number of queries may include both an absolute value and arelative value, e.g., the customer has provided accurate responses to atleast 95% of all queries and has been queried at least 50 times.

In some embodiments, the method may include receiving, from at least asecond of the plurality of sensors associated with the retail store,second sensed data indicating a selection of a second product by thecustomer. The second sensed data indicating the selection of the secondproduct may be performed similarly to the method described above inconnection with receiving the first sensed data indicating the selectionof the first product by the customer. It is noted that the designations“first product” and “second product” do not necessarily require thatthese are the first and second products selected by the customer whilein the retail store, though that is a possible designation. For example,the “first product” may be the fifth product selected by the customerand the “second product” may be the tenth product selected by thecustomer. In some embodiments, the second product may be selected apredetermined number of products after the customer selects the firstproduct. For example, the second product may be the fifth product afterthe customer selects the first product. As another example, the firstproduct may be the fifth product selected by the customer and the secondproduct may be the next product selected by the customer (i.e., thesixth product selected by the customer). In some embodiments, the secondproduct may be selected a predetermined period of time after thecustomer selects the first product. For example, the second product maybe selected two minutes after the first product even if the customer hasselected other products in the two-minute period.

In some embodiments, the method may include determining, based on theintegrity indicator, at least one action to take in order to determine asecond property associated with the second product. In some embodiments,different actions may be taken based on whether the integrity indicatorindicates that the first information received from the customer isaccurate or inaccurate. Some of the different actions that may be takenare described in further detail below. In some embodiments, the methodmay include initiating the at least one action.

In some embodiments, the second property may be indicative of a producttype associated with the second product. In some embodiments, theproduct type associated with the second product may include a broadcategory of the product type, e.g., a bottle of soda, a loaf of bread, anotepad, or the like.

In some embodiments, the second property may be indicative of a quantityassociated with the second product. For example, the second property mayindicate that the customer selected a six-pack of soda cans or a packageof three cans of vegetable soup. In one example, the second property mayindicate whether the customer selected a first number of units or asecond number of units (for example, five or six, one or two, etc.) ofthe second product. In one example, the second property may indicate aweight of the second product. For example, the second product may be acustom-made product that may have different weights, and the secondproperty may indicate a weight of the second product. In one example,the second property may indicate a volume of the second product. Forexample, the second product may be a custom-made product that may havedifferent volumes, and the second property may indicate a volume of thesecond product.

In some embodiments, the second property may be indicative of a priceassociated with the second product. For example, the second property maybe the “shelf price” of the second product, where the “shelf price” isthe price of the second product as posted on the retail shelf. In someembodiments, the second property may be a discounted price of the secondproduct if, for example, the second product is currently being sold at adiscount by the retail store. In some embodiments, the second propertymay be a discounted price of the second product if, for example, thecustomer is identified as a “frequent shopper” at the retail store(e.g., via a retail store loyalty program) and the retail store providesa discounted price to customers identified as “frequent shoppers.”

In some embodiments, the first property and the second property mayrelate to a common product characteristic. For example, the firstproperty and the second property may both be the product type associatedwith the first product and the second product, the quantity associatedwith the first product and the second product, or the price associatedwith the first product and the second product.

In some embodiments, the first property and the second property mayrelate to different product characteristics. For example, the firstproperty may be the quantity associated with the first product and thesecond property may be the price associated with the second product.Various combinations of product characteristics of the first product andthe second product are possible. The combination of productcharacteristics of the first product and the second product used by themethod does not change the overall operation of the method.

In some embodiments, when the integrity indicator associated with thefirst information received from the customer indicates that the firstinformation is accurate, the determined at least one action includesissuing a second query to the customer, receiving second informationfrom the customer in response to the second query, and determining thesecond property based on the second information received from thecustomer. In some embodiments, the second query may be issued to thecustomer in a similar manner as the first query described above. In someembodiments, the second information may be received from the customer ina similar manner as the first information described above. In someembodiments, determining the second property may be performed in asimilar manner as determining the first property as described above.

In some embodiments, at least one of the first query or the second queryis provided to the customer via a personal mobile communication deviceof the customer. In some embodiments, at least one of the first query orthe second query is provided to the customer via a communication devicephysically connected to a shopping cart.

In some embodiments, the second information may be received from thecustomer in a similar manner as the first information described above,e.g., via the customer's mobile device or via the communication deviceconnected to the shopping cart. As discussed above, the second propertymay be indicative of a product type associated with the second product,a quantity associated with the second product, or a price associatedwith the second product.

In some embodiments, when the integrity indicator associated with thefirst information received from the customer indicates that the firstinformation is inaccurate, the determined at least one action mayinclude requiring the customer to scan a barcode associated with thesecond product. If the integrity indicator associated with the firstinformation indicates that the first information was inaccurate (i.e.,that the customer may not be trustworthy), then the method may requirethe customer to perform an action to verify the second property of thesecond product. In some embodiments, the customer may scan the barcodeassociated with the second product using the customer's mobile device.In some embodiments, the customer may scan the barcode associated withthe second product using the communication device connected to theshopping cart. In some embodiments, the customer may scan the barcodeassociated with the second product using a sensor (e.g., a barcodescanner) physically connected to the shopping cart.

In some embodiments, when the integrity indicator associated with thefirst information received from the customer indicates that the firstinformation is inaccurate, the determined at least one action mayinclude requiring the customer to capture an image of the secondproduct. In some embodiments, the customer may capture the image of thesecond product using the customer's mobile device. In some embodiments,the customer may capture the image of the second product using thecommunication device connected to the shopping cart. In someembodiments, the customer may capture the image of the second productusing an image sensor physically connected to the shopping cart.

In some embodiments, when the integrity indicator associated with thefirst information received from the customer indicates that the firstinformation is inaccurate, the determined at least one action mayinclude causing a store associate associated with the retail store toconfirm the second property associated with the second product. In someembodiments, the store associate may confirm the second propertyassociated with the second product using the customer's mobile device.In some embodiments, the store associate may confirm the second propertyassociated with the second product using the communication deviceconnected to the shopping cart. In some embodiments, the store associatemay confirm the second property associated with the second product usinga handheld device of the store associate (e.g., capturing device 125D).

In some embodiments, when the integrity indicator associated with thefirst information received from the customer indicates that the firstinformation is inaccurate, the determined at least one action mayinclude requiring the customer to engage in a non-frictionless checkoutprocess. In some embodiments, the non-frictionless checkout process mayinclude the customer going to a checkout station to complete thecheckout process. In some embodiments, the checkout station may be“self-service,” such that the customer scans the products in theshopping cart. In some embodiments, the checkout station may be staffedby a store associate who may assist the customer to complete thecheckout process. For example, the store associate may scan the productsin the shopping cart. In some situations, the customer and the storeassociate may each scan a portion of the products selected by thecustomer. In some embodiments, the store associate may scan the productsin the shopping cart using a handheld device. In some embodiments, thecustomer may unload the products from the shopping cart to a conveyorbelt and the store associate may scan the products using a scanningdevice positioned adjacent to the conveyor belt (i.e., a “traditional”checkout station).

In some embodiments, the method may include analyzing the second senseddata to determine a plurality of alternative plausible values for thesecond property associated with the second product. In some embodiments,the second sensed data may be sent to a remote server for analysis. Insome embodiments, the method may include further basing the selection ofthe at least one action on the determined plurality of alternativeplausible values.

In some embodiments, the second sensed data may be used to determinepossible values of the second property and then the selected action maybe based on the possible values to determine what the second propertyis. For example, if a shape of a container is analyzed in an image, andthe shape appears to be a container of a certain size and the senseddata may also be analyzed to determine that the customer is in an aislein the retail store with refrigerated shelving units or milk products,the selected action may be to query the customer, “Did you select agallon of milk?” The selected action may be less specific if the senseddata reveals fewer details. For example, if all that is known about thecontainer is the shape (e.g., the container may include milk, water, orsoda), the selected action may be to ask the customer to pick theproduct the customer selected from a list of drink categories. In someembodiments, more than one selected action may be needed to refine whatthe second property is if the sensed data is limited (e.g., issue afirst query whether the customer selected a beverage and, if so, issue asecond query to ask if the customer selected one of the beverages on alist).

In some embodiments, the method may include analyzing the second senseddata to determine a plurality of likelihoods, each likelihood of theplurality of likelihoods is associated with a different alternativeplausible value for the second property associated with the secondproduct. In one example, each likelihood may represent a probabilitythat the plausible value for the second property is the second property.In another example, each likelihood may represent a degree of likelihood(such as ‘high’, ‘low’, etc.) that the plausible value for the secondproperty is the second property. In some examples, the likelihood mayrepresent uncertainty in the determination of the second propertyassociated with the second product. For example, data from a pluralityof sensors may be analyzed, and analysis of data from different sensorsmay suggest different values for the second property. In this example,the likelihood associated with a particular plausible value for thesecond property may be determined based on a prior confidence in thesensor corresponding to the particular plausible value, may bedetermined based on the number of sensors corresponding to theparticular plausible value, and so forth. In another example, the secondsensed data may be analyzed using a plurality of algorithms (forexample, a plurality of trained machine learning models, each trainedusing a different training set), and different algorithms may suggestdifferent values for the second property. In this example, thelikelihood associated with a particular plausible value for the secondproperty may be determined based on a prior confidence in the trainedmachine learning model corresponding to the particular plausible value,may be determined based on the number of trained machine learning modelscorresponding to the particular plausible value, and so forth. In someembodiments, each likelihood may be represented by a numerical value.For example, a value between 0 and 1, a positive number, a real number,and so forth. In some embodiments, the method may include further basingthe selection of the at least one action on a distribution of thedetermined plurality of likelihoods. For example, the selection of theat least one action may be based on a variance or a standard deviationof the distribution. In another example, the selection of the at leastone action may be based on a histogram of the distribution. In yetanother example, the selection of the at least one action may be basedon any parameter of the distribution. For example, in response to aparticular parameter of the distribution having a first value, a firstaction may be selected, and in response to the particular parameter ofthe distribution having a second value, a second action may be selected,the second action may differ from the first action. In some examples, amachine learning model may be trained using training examples to selectactions based on distributions. An example of such training may includea sample distribution, together with a label indicating a desiredselection of actions in response to the sample distribution. The trainedmachine learning model may be used to analyze the distribution of thedetermined plurality of likelihoods and select the at least one action.Some non-limiting examples of such a distribution of the determinedplurality of likelihoods may include a discrete distribution, acontinuous distribution, a normal distribution, a finite distribution,an infinite distribution, a Yule-Simon distribution, a Polya-Eggenbergerdistribution, and so forth. In some embodiments, the plausible value forthe second property having a highest likelihood may be used to selectthe at least one action.

In some embodiments, the second sensed data may be used to determine aprobability (i.e., likelihood) of several different possible values ofthe second property and then the selected action may be based on theprobability distribution to determine what the second property is. Forexample, if a shape of a container is analyzed in an image, and based onthe probabilities, the shape may be determined to be a container of acertain size. The sensed data may also be analyzed to determine that thecustomer is in an aisle (in some embodiments, the customer's locationwithin the retail store may be based on a probability that the customeris in a particular aisle of the retail store) with refrigerated shelvingunits or milk products, the selected action may be to query thecustomer, “Did you select a gallon of milk?” The selected action may beless specific if the sensed data reveals fewer details. For example, ifall that is known about the container is the shape (e.g., the containermay include milk, water, or soda), the selected action may be to ask thecustomer to pick what product the customer selected from a list of drinkcategories. In some embodiments, more than one selected action may beneeded to refine what the second property is if the sensed data islimited. For example, a first action may include issuing a first querywhether the customer selected a beverage and, if so, a second action mayinclude issuing a second query to ask if the customer selected one ofthe beverages on a list.

In some embodiments using the preceding example, the answer optionsassociated with the first query and the second query may be based on theprobability distribution. For example, if the probability distributionindicates that the customer is most likely to have selected milk ororange juice (as opposed to soda), then the answer options may belimited to “milk” and “orange juice.” In some embodiments, the answeroptions associated with the first query and the second query may beincluded if the probability associated with the answer option exceeds athreshold (e.g., greater than 10%). For example, if the probabilitydistribution indicates a 50% probability that the customer selectedmilk, a 45% probability that the customer selected orange juice, and a5% probability that the customer selected soda, the 5% probability maybe below the threshold and the answer options may then be limited to“milk” and “orange juice.”

FIG. 21A is an exemplary screenshot 2100 of a customer query displayedon a display attached to shopping cart, consistent with the presentdisclosure. The screenshot 2100 includes a query 2102 relating to aproduct selected by the customer. In some embodiments, query 2102 mayrelate to the first product or the second product selected by thecustomer. A first answer choice 2104 and a second answer choice 2106 arepresented as possible responses to query 2102. Selecting first answerchoice 2104 or second answer choice 2106 provides the first informationfrom the customer. As discussed above, the number of answer choicespresented to the customer in response to query 2102 may vary accordingto the size of the display.

As an example, if the customer selected a can of crushed tomatoes, query2102 may ask the customer to confirm the size of the can of crushedtomatoes, “What size is the can of crushed tomatoes?” Based on query2102, the displayed answer choices may relate to possible answers toquery 2102. As shown in FIG. 21A, first answer choice 2104 is a firstpossible size for a can of crushed tomatoes (14 ounce) and second answerchoice 2106 is a second possible size for the can of crushed tomatoes(28 ounce). In some embodiments, if multiple answer choices aredisplayed, one or more of the displayed answer choices may not be avalid answer choice (e.g., a 20 ounce can of crushed tomatoes, when sucha size can does not exist). In some embodiments, if multiple answerchoices are displayed, one or more of the displayed answer choices maybe close to another answer choice (e.g., one answer choice is “27 ounce”and a second answer choice is “28 ounce”). In some embodiments,providing answer choices that are not valid or are close to existinganswer choices, may enable different ways to determine the customer'sintegrity.

FIG. 21B is a top plan view of a shopping cart 2120, including one ormore sensors and a display screen attached to the shopping cart,consistent with the present disclosure. Shopping cart 2120 includes adisplay 2122 connected to a handle 2124. An interior 2126 of shoppingcart 2120 includes a bottom having one or more sensors 2128 connectedthereto. In some embodiments, display 2122 may be connected to adifferent portion of shopping cart 2120 such that display 2122 isvisible to a customer while pushing the shopping cart via handle 2124.In some embodiments, sensors 2128 may include a single large sensor(e.g., a weight pad) or one or more individual sensors. In someembodiments in which individual sensors are used, the sensors may have auniform size and may be arranged in a pattern (e.g., a grid) to enablesensing of a majority of the bottom of shopping cart 2120. In someembodiments in which individual sensors are used, the sensors may varyin size and may be arranged in a pattern (e.g., a grid) to enablesensing of a majority of the bottom of shopping cart 2120. In someembodiments, shopping cart 2120 may have a different size or/shape thanthe example shown in FIG. 21B. For example, as shown in FIG. 21B,interior 2126 of shopping cart 2120 may have a trapezoidal shape. Inother examples, interior 2126 of shopping cart 2120 may be rectangular,square, or other quadrilateral shape.

FIG. 22A is an exemplary screenshot 2200 of a customer query displayedon the customer's mobile device, consistent with the present disclosure.The screenshot 2200 includes a query 2202 relating to a product selectedby the customer. In some embodiments, query 2202 may relate to the firstproduct or the second product selected by the customer. In the exampleshown in FIG. 22A, query 2202 asks the customer to take a picture of theproduct (e.g., a can of tomatoes) with the customer's mobile device. Insome embodiments, the camera of the customer's mobile device may beautomatically activated and an image capture area 2204 may be shown toenable the customer to align the picture of the product within the imagecapture area 2204. Image capture area 2204 is optional, and in someembodiments, image capture area 2204 may not be shown on screenshot2200. In some embodiments, the camera of the customer's mobile devicemay be activated with the customer's permission. In some embodiments,the customer's permission may be generally provided for a particularapplication running on the customer's mobile device (e.g., anapplication issued in connection with the retail store) to enable cameraaccess by the application. In some embodiments, the customer'spermission may be required each time the camera is to be accessed. Insome embodiments, how and whether the camera is accessed may bedetermined by a setting on the customer's mobile device.

FIG. 22B is a schematic illustration of a customer taking a picture of aproduct with the customer's mobile device, consistent with the presentdisclosure. Continuing the example of query 2202, a customer 2220 usesthe customer's mobile device 2222 to take a picture of the product 2224.

Screenshot 2100 shown in FIG. 21A and screenshot 2200 shown in FIG. 22Aare exemplary and as described above, many different queries may beissued to the customer. For example, if display 2122 connected toshopping cart 2120 is configured to include a camera, query 2202 may beissued to the customer via display 2122 and the customer may positionthe product such that an image of the product can be captured by thecamera of display 2122. As another example, query 2102 may be issued tothe customer's mobile device and the customer may select answer choice2104 or 2106 via the mobile device.

FIG. 23 is a flowchart of an exemplary method 2300 for frictionlessshopping based on estimated shopper integrity, consistent with thepresent disclosure. Method 2300 may be performed by at least oneprocessor or at least one processing unit, such as processing unit 130or processing device 202. It is to be understood that throughout thepresent disclosure, the term “processor” is used as a shorthand for “atleast one processor.” In other words, a processor may include one ormore structures that perform logic operations whether such structuresare collocated, connected, or dispersed. In some embodiments, anon-transitory computer readable medium may contain instructions thatwhen executed by a processor cause the processor to perform method 2300.Further, method 2300 is not necessarily limited to the steps shown inFIG. 23 , and any steps or processes of the various embodimentsdescribed throughout the present disclosure may also be included inmethod 2300.

In step 2302, method 2300 receives, from at least a first of a pluralityof sensors associated with a retail store, first sensed data indicatinga selection of a first product by a customer. The plurality of sensorsmay include any type of sensors described herein, such as an imagesensor (e.g., as a part of an image capture device such as image capturedevice 506), a motion sensor, a sound sensor, a proximity sensor, atemperature sensor, a biometric sensor, a pressure sensor, a weightsensor, a light sensor, a resistive sensor, a capacitive sensor, aninductive sensor, a vacuum pressure sensor, a high pressure sensor, aconductive pressure sensor, an infrared sensor, a photo resistor sensor,a photo-transistor sensor, a photo-diode sensor, an ultrasonic sensor,or the like. The plurality of sensors may be placed in multiplelocations throughout the retail store (e.g., on each retail shelf orproduct display) and/or be placed in a shopping cart used by a customer(e.g., the sensors may be physically connected to the shopping cart).

In step 2304, method 2300 determines, from the first sensed data, afirst property associated with the first product. In some embodiments,the property associated with the first product may include one or morecharacteristics of the product, such as one or more visually discernablefeatures attributed to the product or one or more features that may bedetected by one or more of the plurality of sensors (e.g., size of theproduct, shape of the product, information on a label associated withthe product, weight of the product, etc.).

In step 2306, method 2300 issues a first query to the customer withrespect to the first property associated with the first product. Thefirst query may be a “test event” to estimate the customer'strustworthiness based on the received first information. For example,the first query may be displayed to the customer via screenshot 2100(shown in FIG. 21A) or screenshot 2200 (shown in FIG. 22A).

In step 2308, method 2300 receives first information from the customer.The first information may include the customer's response to the firstquery. For example, in response to query 2102 (shown in FIG. 21A), thefirst information may include the customer's answer choice in responseto query 2102 (e.g., first answer choice 2104 or second answer choice2106). As another example, in response to query 2202 (shown in FIG.22A), the first information may include the image of the product takenby the customer.

In step 2310, method 2300 compares the first information received fromthe customer with the first property associated with the first product.The comparing may be performed by any known comparison means.

In step 2312, method 2300 determines, based on the comparison betweenthe first information received from the customer with the first propertyassociated with the first product, an integrity indicator associatedwith the customer. The integrity indicator may be any type of value thatmay be used to track a characteristic of the customer.

In step 2314, method 2300 receives, from at least a second of theplurality of sensors associated with the retail store, second senseddata indicating a selection of a second product by the customer. Thesecond sensed data indicating the selection of the second product may beperformed similarly to receiving the first sensed data indicating theselection of the first product by the customer.

In step 2316, method 2300 determines, based on the integrity indicator,at least one action to take to determine a second property associatedwith the second product. In some embodiments, different actions may betaken based on whether the integrity indicator indicates that the firstinformation received from the customer is accurate or inaccurate.

In step 2318, method 2300 initiates the at least one action asdetermined in step 2316.

Digital activities of users may reveal information about dynamicallychanging situations in a retail store. Conventional methods foroverseeing retail stores, such manually inspecting aisles, ordesignating one or more human supervisors to physically circuit througheach region of a retail store, may be costly and impractical,particularly for large stores with many aisles and departments.Information gleaned from digital activities of users while in theretails store may thus provide convenient and timely updates on thestatus or condition of areas in the retail store and may facilitate thestore to promptly address situations that require attention in a costeffective and efficient manner. For example, customers may post updateson social media or share images with friends when encountering amispriced or mislabeled product, a mishap (e.g., a spill), a promotion,a delay (e.g., due to long checkout queues or shelf restocking),availability of a product, and the like. Technology may be provided tomonitor digital activities in a retail store and cluster such activitiesto identify patterns associating particular areas of the retail storewith a given cluster of online activity. The cluster may be furtheranalyzed to troubleshoot a problem in the particular area. Informationpublished on digital platforms (e.g., online digital activities) fromwithin the retail store may thus allow the store to automaticallyalleviate problems efficiently and accurately.

According to some embodiments, a method may be implemented to associateonline digital activities with conditions in a retail store. The methodmay include detecting a plurality of online digital activitiescorresponding to subject matter related to a retail store. For example,a plurality of customers may post images or videos captured in theretail store on a social media account and may tag the post with ahashtag identifying the retail store. The plurality of online digitalactivities may originate from customers in the retail store. Forexample, the customers may be located in the vicinity of a particularcondition. At least one processor may receive position informationindicative of positions in the retail store from which each of thedetected plurality of online digital activities originated. The at leastone processor may cluster the detected online digital activities basedon the received position information to provide one or more clusters ofonline digital activities. Each of the one or more clusters may beassociated with a different region of the retail store. Further, animage acquisition device may capture one or more images of a particularregion of the retail store associated with a particular cluster ofdetected online digital activities. The captured images may be analyzed(e.g., by at least one processor) to detect a condition of the retailstore associated with the particular cluster of detected online digitalactivities. At least one remedial action may be automatically initiatedto address the detected condition.

FIGS. 24A and 24B, together, illustrates an exemplary implementation ofa system 2400 for associating online digital activities with conditionsin a retail store, consistent with some disclosed embodiments. System2400 includes at least a processing unit 2404 (e.g., one or more ofimage processing unit 130 of FIG. 1 ), at least one sensor 2406 (e.g.,corresponding to at least one of sensors 125A, 125B, 125C of FIG. 4A)and a communications network 2408 (e.g., communications network 150 ofFIG. 1 ). In some embodiments, processing unit 2404 may include one ormore of servers 135 and one or more databases 140 of FIG. 1 . At leastone sensor 2406 may include one or more detectors to detect signals,such as radio signals (e.g., WiFi, BlueTooth, Zigbee, RFID signals),optical signals (e.g., visible light as images and/or video, infrared,and/or lidar signals), acoustic signals (e.g., ultrasound and/or audio),vibrations signals, and/or any other means for detection within theretail store. Sensor 2406 may be associated with a unique identifierallowing processing unit 2404 to associate data received from sensor2406 with a position in the retail store, such as a specific aisle,counter, or any other specific region within the retail store.Processing unit 2404 may receive sensed data captured by sensor 2406 viacommunications network 2408. Communications network 2408 mayadditionally be communicatively coupled to cloud servers 2410, 2412, and2414, respectively, via any combination of wired and/or wirelesscommunications means. Cloud servers 2410, 2412, and 2414 may beassociated with one or more digital platforms (e.g., social networks)configured for online digital activities via communications networks2416, 2418, and 2420, respectively. Although cloud servers 2410, 2412,and 2414 may each be associated with a different digital platform and/orcommunications network, this is not required. While each of customers2422, 2424, and 2426 is shown associated with a different communicationsnetwork and cloud server, this is not intended to be limiting, and twoor more customers may conduct their online activity via the same cloudserver and/or communications network.

Customers 2422, 2424, and 2426 may use mobile devices 2428, 2430, and2432 while in the retail store to perform online digital activities viacloud servers 2410, 2412, and 2414 and communications networks 2416,2418, and 2420, respectively. For example, customers 2422, 2424, and2426 may acquire images of a condition 2402 (e.g., a spill requiringclean-up or maintenance) via mobile devices 2428, 2430, and 2432, andshare the acquired images on one or more digital platforms associatedwith cloud servers 2410, 2412, and 2414, respectively. Customers 2422,2424, and 2426 may share the acquired images in association withadditional data, such as location data, a time stamp, text content suchas a hashtag associated with the retail store, or any other data thatmay provide a way to associate the online digital activities ofcustomers 2422, 2424, and 2426 with condition 2402 in the retail store.Processing unit 2404 may continually monitor online digital activities,e.g., occurring on the online digital platforms associated with cloudservers 2410, 2412, and 2414, and thereby detect the online digitalactivities performed by customers 2422, 2424, and 2426 in associationwith condition 2402. Processing unit 2404 may analyze the detectedonline digital activities and determine a correlation between the onlinedigital activities, e.g., based on one or more common characteristicsdetermined by analyzing the online digital activities, such as bycomparing location data, a time stamp, text content, image content, orany other digital content indicating a correlation between the onlinedigital activities of customers 2422, 2424, and 2426. Based on theanalysis of the online digital activities, processing unit 2404 maycluster the online digital activities of customers 2422, 2424, and 2426,e.g., in relation to condition 2402.

Additionally and/or concurrently, sensor 2406 affixed to a display area2466 in proximity to condition 2402 may sense data relating to condition2402. For example, sensor 2402 may continually acquire images of theregion surrounding display area 2466, and thereby acquire images relatedto condition 2402. Optionally, sensor 2402 may acquire images ofcustomers 2422, 2424, and 2426 performing the online digital activitiesrelating to condition 2402. Processing unit 2404 may receive the datasensed by sensor 2402 in association with the unique identifier ofsensor 2402 and thereby associate data received from sensor 2402 withthe region surrounding display area 2466.

Based on the clustering of the online digital activities of customers2422, 2424, and 2426 in relation to condition 2402 and the data receivedby sensor 2402, processing unit 2404 may detect condition 2402 as asituation that needs to be addressed. In response to the detection,processing unit 2404 may initiate an automated remedial action, such asto command via communications network 2408 a cleaning robot 2462 (FIG.24B) to remedy condition 2402. Thus, the monitoring of the onlinedigital activities of customers 2422, 2424, and 2426 while in the retailstore, may facilitate the detection of condition 2402 by processing unit2404. Consequently, condition 2402 may be addressed conveniently andefficiently by automatically initiating a remedial action for condition2402.

Some embodiments involve a non-transitory computer-readable mediumincluding instructions that when executed by a processor cause theprocessor to perform a method for associating online digital activitieswith conditions in a retail store. The term “processor” may include oneor more hardware processors (e.g., processing device 202 of FIG. 2 )and/or emulated computing processors configured to operate in adistributed manner. The term “online” may refer to an association with apublic and/or private computer network, such as an intranet, virtualprivate network, and the Internet. The term “online digital activities”may refer to actions or activities performed using at least one computer(e.g., digital activities) that is connected to at least one othercomputer via a network, e.g., the Internet. For example, an onlinedigital activity may include any of publishing, authoring, editing,digitally reacting to (e.g., commenting on, liking or disliking a post),or sharing digital content (e.g., authored by others), such as text,image, audio or video content on a digital platform, such as a socialmedia platform. The term “retail store” may refer to an establishmentoffering products for sale, as described earlier. The term “conditionsin a retail store” may refer to a state, situation, or set ofcircumstances in a retail store that may affect a shopping experience ofa customer in the retail store. The condition in the retail store mayrelate to an item or object, (e.g., a product for restocking orrepricing, a cart, a ladder or stool, a cleaning machine, or robot), anissue (e.g., a price, quantity, availability, shortage, promotion,exhibit or display for a product, a bottleneck at a counter or checkoutqueue, a number of functioning or malfunctioning counters and/orcheckout queues, or an accessibility of an area within the retailstore), a problem (e.g., a hazard, broken glass, danger, fire, spill,and obstruction due to a shopping or loading cart, or topped display),or any other matter that may affect a shopping experience and/orlocation in the retail store.

As shown in FIG. 24A, a retail store may have a condition 2402, such asa spill, or hazard obstructing a shopping aisle of the retail store.Customers 2422, 2424, and 2426, may be aware of condition 2402, such asby observing (e.g., by seeing, hearing, smelling, tasting, feeling,sensing or gaining awareness) condition 2402 or via any othercommunicating means, such as by receiving a notification, message,image, video, or phone call about condition 2402. Customers 2422, 2424,and 2426 may each perform one or more online digital activities viamobile devices 2428, 2430, and 2432, for example by publishing a postrelated to condition 2402 via cloud servers 2410, 2412, and 2414 andcommunications networks 2416, 2418, 2420, respectively. At least some ofthe online digital activities may be associated with condition 2402 inthe retail store.

Accordingly, some embodiments involve detecting a plurality of onlinedigital activities including subject matter related to a retail store.The term “detecting a plurality of online digital activities” may referto recognizing, identifying, or otherwise determining a plurality ofonline digital activities. The detection may be implemented via ahardware device (e.g., sensor, or peripheral device), via software(e.g., an analysis of data received from one or more hardware device),or a combination of hardware and software detection. For example, thedetecting may involve scanning, searching, or crawling social mediaplatforms for location data, images, hashtags, names, identifiers, ortextual content associated with the retail store. The term “subjectmatter related to a retail store” may include textual content (e.g., aname, phrase, or hashtag), metadata (e.g., location data, time,identifying tags, etc.), visual content (e.g., an image or video) audiocontent (e.g., a voice recording, song, melody, jingle, etc.) or anyother information at least partially associated with the retail store.The association may be based on any aspect of the retail store, such asa product, brand, promotion, event, time period, display, price,inventory (e.g., availability or shortage) access, delay, queue (e.g.,insufficient or malfunctioning queue), a physical aspect (e.g., aparticular shelf, floor area, door, or aisle, an issue with lighting,heating, or cooling), a safety hazard (e.g., smoke, broken glass,spill), an obstruction (e.g., loading cart, collapsed product display),and any other issue pertaining to the retail store. The subject mattermay assist in differentiating between online digital activities that arerelated to (e.g., associated with) the retail store versus onlinedigital activities that are unrelated to the retail store. For example,a plurality of shoppers may each upload a post to a social media accountwhile inside the retail store. Some posts may be unrelated to the retailstore, while some posts may include subject matter related to the retailstore, e.g., as a hashtag identifying the retail store, visual or audiocontent (e.g., an image or video) captured from the interior of theretail store, location data corresponding to the location of the retailstore, and any other digital content related to the retail store. Theposts may be uploaded from inside the retail store via a wirelesscommunication network (e.g., a mobile or cellular communicationsnetwork, radio such as WiFi, BlueTooth, Zigbee, infrared, microwavecommunication, satellite communication) and one or more cloud servers.At least one processor may monitor (e.g., by executing a softwareapplication) one or more digital platforms for digital contentpotentially containing subject matter related to the retail store.

FIGS. 24A, 24B, and 25 together, illustrate an exemplary implementationfor detecting a plurality of online digital activities including subjectmatter related to a retail store, consistent with some disclosedembodiments. Referring to FIG. 24A, on becoming aware of condition 2402,customers 2422, 2424, and 2426 may each upload a post on one or moresocial media platform via cloud servers 2410, 2412, and 2414 andcommunications networks 2416, 2418, 2420, respectively. For example,processing unit 2404 may execute a software application that monitors(e.g., by continually crawling, scanning, querying, or searching) thedigital platforms on cloud servers 2410, 2412, and 2414 for content(e.g., online digital activities) associated with the retail store. Asone example, processing unit 2404 may search for content includinglocation data corresponding to the location of the retail store, togenerate a listing of online digital activities. Referring to FIG. 25 ,an exemplary listing of online digital activities 2440 is shown fordigital platforms 2442, 2444, and 2446, e.g., associated with cloudservers 2410, 2412, and 2414 (FIGS. 24A and 24B), respectively. Listing2440 may include posts 2450 and 2452 published on platform 2442, posts2454 and 2456 published on platform 2444, and posts 2458 and 2460published on platform 2460. Posts 2450 to 2460 include location dataassociated with the retail store (e.g., “Superstore, Anytown, U.S.A.”)and may thus include subject matter potentially related to the retailstore. The use of location data to detect the online digital activitiesis intended as exemplary only, and other identifying information may beused as well or instead. Processing unit 2404 may detect posts 2450 to2458 as online activities that include subject matter related to theretail store, e.g., based on the inclusion of the hashtag #SuperStore,and may determine that post 2460 lacks subject matter related to theretail store, e.g., based on an absence of digital content associatedwith the retail store.

Accordingly, the plurality of online digital activities may originatefrom customers in the retail store. The term “originate” may refer to asource for the online digital activity, such as the creator or author ofdigital content associated with the online digital activity, aphotographer or videographer that captured an image or video associatedwith the online digital activity, or any other individual (or robot)associated with creating, authoring, sharing or uploading digitalcontent associated with the online digital activities. Thus, onlinedigital activities originating from customers inside the retail storemay be distinguished from online digital activities originating fromusers outside the retail store.

While in the retail store, customers 2422, 2424, and 2426 (FIG. 24A) maypublish digital content 2452, 2454, and 2458 (FIG. 25 ) to digitalplatforms 2442, 2444, and 2446 via cloud servers 2410, 2412, and 2414and communications networks 2416, 2418, 2420, respectively. For example,digital content 2442, 2446, and 2448 may include any of location datafor the retail store, a hashtag for the retail store, image data fromwithin the retail store, and any other data identifying the retailstore, to indicate the origination of digital content 2442, 2446, and2448 by customers 2422, 2424, and 2426, respectively inside the retailstore.

Some embodiments may involve receiving position information indicativeof positions in the retail store from which each of the detectedplurality of online digital activities originated. The term “positioninformation” may refer to one or more of position, orientation, orlocation (e.g., geospatial) data such as may be acquired via one or moreof GPS networks, cellular networks (e.g., 4G, 5G), RFID trackers, radio(e.g., radar), infrared (e.g., LIDAR), optic (e.g., cameras), oracoustic (e.g., ultrasound) sensors, inertial measuring units (IMUs),and any other technological means for acquiring position, location andorientation data. For example, a mobile device may be equipped with aGPS receiver for transmitting tracking data to a server associated witha digital platform. When the user of the mobile device performs anonline digital activity on the digital platform, the server may includethe tracking data, allowing the server to associate the online digitalactivity with the current location of the user. A computer associatedwith the retail store may receive the position information in responseto querying the digital platforms for digital content related to theretail store. Additionally, or alternatively, one or more sensors (e.g.,optic cameras, LIDAR sensors, ultrasound sensors, RFID sensors)positioned in the retail store may provide position informationindicating positions in the retails store from which the online digitalactivities originated. For example, the position information may includeacquired image data, and the method may further include analyzing theacquired image data to determine the positions in the retail store fromwhich each of the detected plurality of online digital activitiesoriginated. The image data may be acquired by one or more cameraspositioned in the retail store, each associated with a unique identifierthat associates each image with a position in the retail store. Theimages may be transmitted with the unique identifier (e.g., as metadata)to a computer for analysis to identify one or more features in theimages corresponding to the online digital activity, allowing todetermine positions from which each of the online digital activitiesoriginated.

In some embodiments, processing unit 2404 may query cloud servers 2410,2412, and 2414 for digital content associated with the retail store(e.g., by searching on location data). In response to the query,processing unit 2404 may receive position information (e.g., GPS,cellular, WiFi, BlueTooth, Zigbee location data) with each of posts 2450to 2460. The position information may indicate positions within theretail store from which the detected online digital activitiesoriginated. Additionally, or alternatively, processing unit 2404 mayreceive the position information from sensor 2406. For example, sensor2406 may capture images, e.g., of condition 2402 and customers 2422,2424, and 2426 as location data. Additionally, or alternatively, sensor2406 may include a beacon for sensing the locations of mobile devices2428, 2430, 2432 as a radio signal. Sensor 2406 may transmit the sensedposition information to processing unit 2404, which may analyze theposition information (e.g., as images and/or radio signals) to detectcustomers 2422, 2424, and 2426 and condition 2402. Processing unit 2404may compare condition 2402, detected via sensor 2406, with the digitalcontent included in posts 2452, 2454, and 2458, and determine that thereceived position information indicates positions in the retail storefrom which posts 2452, 2454, and 2458 originated. In a similar manner,processing unit 2404 may determine that the captured images receivedfrom sensor 2406 do not indicate a position in the retail store fromwhich post 2460 originated.

According to some embodiments, the position information may furtherindicate at least a portion of a path traversed by at least one customerin the retail store. For example, a customer may be tracked whileshopping in the retail store. The tracking may be performed using datareceived from any of one or more sensors (e.g., image, infrared,acoustic, WiFi, BlueTooth, Zigbee) positioned in the retail store, e.g.,on a shelf, ceiling, wall, shopping cart, or any other object in theretail store. Additionally, or alternatively, the customer may betracked via an application installed on a mobile device. The trackingdata may enable determining at least a portion of a path traversed bythe customer in the retail store. For example, mobile device 2412 ofcustomer 2422 may be configured with a tracking application that trackspath 2480 of customer 2422. Additionally, sensor 2406 may track the pathof customer 2422. Processing unit 2404 may receive data indicative ofpath 2464 from the tracking application and/or sensor 2406 as positioninformation for customer 2422.

According to some embodiments, the position information may be generatedby one or more mobile devices. The mobile devices may be personaldevices, e.g., carried or worn by the customers in the retail store.Additionally or alternatively, one or more mobile devices may belong tothe retails store, e.g., configured with a shopping cart or bag or anautomated drone or robot operating in the retail store. In someembodiments, mobile devices 2428, 2430 and 2432 may each be configuredwith a software application to track customers 2422, 2424, and 2426, inthe retail store respectively. The software application may transmit thetracking data to processing unit 2404 as position information. Thus,referring to FIG. 24A, one or more of mobile devices 2628, 2630 and 2632may be configured with a software application to track any of customers2622, 2624, and 2426, in the retail store, respectively. The softwareapplication may transmit tracking data to processing unit 2404 asposition information via communications network 2408.

According to some embodiments, the position information may furtherindicate an orientation of the one or more mobile devices duringgeneration of one or more of the plurality of online digital activities.For example, one or more of the mobile devices may be provided with oneor more of an inertial measuring unit (IMU), e.g., including agyroscope, compass, or accelerometer for determining the orientation andposition of the mobile device. The orientation of the mobile device mayfacilitate in determining the position of each mobile device withrespect to the condition. For example, each of mobile devices 2428,2430, and 2432 may include an IMU detecting the orientation and positionof mobile device, e.g., with respect to condition 2402. Processing unit2404 may receive the IMU data from each of mobile devices 2428, 2430,and 2432 and determine that customer 2422 is positioned south-west ofcondition 2402, customer 2424 is positioned north of condition 2402, andcustomer 2426 is positioned south of condition 2402.

According to some embodiments, the position information may include GPScoordinates. For example, one or more of the mobile devices may beprovided with a GPS receiver for receiving location information from oneor more GPS satellites, and determining, e.g., via triangulation, alocation and position of the mobile devices. For example, each of mobiledevices 2428, 2430, and 2432 may include a GPS receiver receiving GPScoordinates from one or more GPS satellites (not shown). Processing unit2404 may receive the GPS coordinates from each of mobile devices 2428,2430, and 2432 to determine the relative positions for each of customers2422, 2424, and 2426.

According to some embodiments, the position information may includenetwork traffic data, and the method may further include analyzing thenetwork traffic data to determine the positions in the retail store fromwhich each of the detected plurality of online digital activitiesoriginated. For example, the retail store may have an internalcommunications network, such as a WiFi network that customers and staffmay use to communicate while inside the retail store. The retail storemay deploy a software application to monitor the traffic on the WiFinetwork to detect a cluster of online digital activity. For example, asudden surge in network traffic may indicate the occurrence of anaccident, special promotion, hazard, or any other condition in theretail store. The WiFi network may include multiple nodes (e.g.,antennas and/or beacons) distributed in the retail store, each having aunique identifier associating each node with a specific area in theretail store. Data traffic sensed by the nodes in the retail store maybe analyzed to detect data traffic surges associated with a specificnode or nodes, thereby associated the traffic with a specific area inthe retail store. The data traffic surge may indicate a cluster ofonline digital activity associated with the specific area and may thusallow determining the positions in the retail store from which eachdetected online digital activity originated.

Some embodiments may involve clustering the detected plurality of onlinedigital activities based on the received position information to provideone or more clusters of detected online digital activities. The term“clustering” may refer to grouping, binning, or otherwise defining acollection based on an association. For example, data may be groupedaccording to a criterion, such as location data, time frame, contenttype, content size, image analysis, text analysis, or any other sharedcharacteristic to form a cluster. The received position information maybe used to group or bin (e.g., cluster) the detected online digitalactivities, for example to detect a pattern or trend. In one example, aspatial clustering algorithm may be used to cluster the detected onlinedigital activities based on the received position information to provideone or more clusters of detected online digital activities. In oneexample, a mathematical graph may be constructed based on the receivedposition information, and a graph cutting algorithm, or a graphclustering algorithm may be used to cluster the detected plurality ofonline digital activities based on the mathematical graph to provide oneor more clusters of detected online digital activities. For example,each node of the mathematical graph may correspond to one of the onlinedigital activities, and a weight of an edge of the mathematical graphconnecting two nodes (e.g., corresponding to two online digitalactivities) may be determined based on at least one of a distancebetween the positions corresponding to the two online digitalactivities, or similarities between subject matter associated with eachof the two online digital activities. In one example, a mathematicalhypergraph may be constructed based on the received positioninformation, and a hypergraph clustering algorithm may be used tocluster the detected plurality of online digital activities based on themathematical hypergraph to provide one or more clusters of detectedonline digital activities. For example, each node of the mathematicalhypergraph may correspond to one of the online digital activities, and aweight of an hyperedge of the mathematical graph connecting three ormore online digital activities may be based on at least one of distancesbetween the positions corresponding to the three or more nodes (e.g.,corresponding to three online digital activities), or similaritiesbetween subject matter associated with the three or more online digitalactivities.

FIG. 25 illustrates an exemplary implementation for clustering thedetected plurality of online digital activities based on the receivedposition information to provide one or more clusters of detected onlinedigital activities, consistent with some disclosed embodiments.Processing unit 2404 (FIG. 24A) may cluster posts 2452, 2454, and 2458based on received position information to provide a first cluster ofdetected online digital activity, e.g., associated with condition 2402.The first cluster is indicated by evenly dashed lines. Similarly,processing unit 2404 may cluster posts 2450 and 2456 into a secondcluster of detected online digital activity, e.g., associated with abakery in the retail store, indicated with uneven dashed lines.

According to some embodiments, each of the one or more clusters may beassociated with a different region of the retail store. For example, onecluster may be associated with a specific aisle, and another cluster maybe associated with a service counter, such as a bakery or butcher. Asshow in FIG. 25 , posts 2452, 2454, and 2458, indicated by evenly dashedbold lines, constitute a first cluster associated with condition 2402 ina specific aisle of the retail store, and posts 2450 and 2456, indicatedby unevenly dashed bold lines, constitute a second cluster associatedwith a bakery of the retail store.

Some embodiments may involve using an image acquisition device tocapture one or more images representative of a particular region of theretail store associated with a particular cluster of detected onlinedigital activities. The term “image acquisition device” may include anydevice configured to acquire image data, such as the capturing devicedescribed earlier. Once a cluster is identified, an image acquisitiondevice may capture one or more images of a particular region in theretail store associated with a particular cluster.

As discussed earlier, sensor 2406 may include one or more cameras forcapturing images of customers 2422, 2424, and 2426 and condition 2402 ata particular region of the retail store, e.g., in proximity to sensor2406. The particular region may be associated with posts 2452, 2454, and2458 (FIG. 25 ) constituting a particular cluster of detected onlinedigital activity. Similarly, referring to FIG. 26 , sensor 2406 mayinclude one or more cameras for capturing images of customers 2622,2624, and 2626 while waiting in a queue or shopping at a particularregion of the retail store (e.g., in a crowded aisle or in proximity tocheckout counters). Images acquired by sensor 2406 may be associatedwith the cluster of digital activities associated with the crowded aisleor queue.

Some embodiments may involve analyzing the one or more captured imagesto detect a condition of the retail store associated with the particularcluster of detected online digital activities. The analyzing may includeimage analysis (e.g., pattern recognition, facial recognition), metadataanalysis (e.g., based on a timestamp and/or location data of the captureimages, image format, type, size) and any other analysis of dataassociated with captured images. The images may be analyzed to detect ashared attribute (e.g., as a cause for the particular cluster ofdetected online digital activities). For example, the images may beanalyzed to detect a spill, obstruction, shortage of products, longqueue, or any other condition that may prompt a cluster of onlinedigital activity in the retail store. In one example, the one or morecaptured images may be analyzed using a visual classification algorithmto classify the one or more images into one of a plurality ofalternative classes. Each alternative class may correspond to adifferent type of condition to thereby detect the condition of theretail store associated with the particular cluster of detected onlinedigital activities. In some examples, the one or more captured imagesmay be analyzed based on the subject matter related to the retail storeand included in the online digital activities corresponding to theparticular cluster to detect a condition of the retail store associatedwith the particular cluster of detected online digital activities. Forexample, a machine learning model may be trained using training examplesto detect conditions from images and/or subject matter information. Anexample of such training example may include a sample image of a sampleretail store and/or a sample subject matter information together with alabel indicating a condition of the sample retail store associated withthe sample subject matter information. The trained machine learningmodel may be used to analyze the one or more captured images based onthe subject matter related to the retail store and included in orassociated with the online digital activities corresponding to theparticular cluster to detect a condition of the retail store associatedwith the particular cluster of detected online digital activities. Insome examples, the online digital activities may be analyzed todetermine the subject matter related to the retail store and included inthe online digital activities. For example, an online digital activitiesmay include communication with a remote system, the remote system may bedetermined (e.g., by an intermediate communication system used for thecommunication with the remote system or by the remote system), and thesubject matter may be determined based on the remote system (e.g., aremote system for online shopping may indicate an online ordering,remote system for online price comparison may indicate an online pricecomparison, and remote system for online nutrition data may indicate anhealth concern). In another example, the online digital activities maybe performed using a software application (or app) associated with theretail store, and the software application may determine the subjectmatter based on the activity in the software application.

In some embodiments, processing unit 2404 may receive images capture bysensor 2406 and analyze the received images to detect condition 2402.Condition 2402 may thus be associated with a particular cluster ofdetected online digital activities, e.g., posts 2452, 2454, and 2458 ofFIG. 25 .

According to some embodiments, the particular region of the retail storemay include one or more aisles of the retail store. In some embodiments,the particular region of the retail store may include a display area fora particular product in the retail store. For example, a first camerapositioned in an aisle may capture one or more images of the aisleassociated with the cluster of online digital activities related to theaisle. A second camera positioned in proximity to the bakery counter maycapture one or more images of a display of baked goods associated withthe cluster of online digital activities related to the bakery.Referring to FIG. 24A, sensor 2406, associated with a particular aisleof the retail store, may capture one or more images of the aisle. Theaisle may include a display area 2434 for a particular product in theretail store.

According to some embodiments, the online digital activities may includeonline digital activities for checking a price of the particularproduct. For example, a customer may inquire about the price of aspecific product sold in the retail store on a digital platform.Referring to FIG. 24A, customer 2422 may post a query for the price of aproduct sold in the retail store on a social media account via cloudserver 2410 and communications network 2416.

According to some embodiments, the online digital activities may includeonline digital activities for ordering the particular product. Forexample, on noticing a shortage of a particular product in the retailstore, a customer may place an order for the product via a siteassociated with the retail store. Referring to FIG. 24A, customer 2422may place an order for a product missing in the retail store on adigital platform via cloud server 2410 and communications network 2416.

According to some embodiments, the detected condition may be associatedwith the particular product. For example, the condition may beassociated with a shortage, spoilage, breakage, promotion, availabilityof the particular product. Referring to FIG. 24A, condition 2402 may beassociated with a particular product that fell and broke, causing aspill.

According to some embodiments, the detected condition may be associatedwith a price of the particular product. For example, a discount,promotion, or dramatic price increase may be subject matter for sharingon a digital platform. Referring to FIG. 25 , post 2450 may beassociated with a price for a particular product (e.g., baked goods)sold in the retail store.

According to some embodiments, the detected condition may be associatedwith an availability of the particular product. For example, a customermay share digital content associated with a shortage of a particularproduct. Referring to FIG. 24A, condition 2402 may be associated with anavailability of a particular product, e.g., that is no longer availableafter falling and breaking.

According to some embodiments, the detected condition may be associatedwith an obstruction in the retail store. For example, a customer mayshare digital content associated with an obstruction, e.g., in an aisleof the retail store. The obstruction may be caused by a spill, breakage,shopping cart, loading cart, special promotion, long queue, and anyother issue that may cause an obstruction in the retail store. Referringto FIG. 24A, condition 2402 may be associated with an obstruction due toa product falling and breaking, causing a spill.

According to some embodiments, the detected condition may be associatedwith a presentation of the particular product. For example, a customermay share digital content associated with how a particular product isdisplayed, e.g., to alert that a product is positioned out of reach ofthe customer or to give feedback on the visual esthetics of the display.Turning to FIG. 24A, condition 2402 may be associated with apresentation of product on display area 2434, such as the height wherethe product is positioned on a shelf, e.g., to alert an employee thatthe particular product is out of reach, located on an incorrect shelf,or located in an incorrect aisle.

According to some embodiments, the detected condition may be associatedwith a potential hazard in the retail store. For example, a customer mayshare digital content associated with a hazard in the retail store.Referring to FIG. 24A, condition 2402 may be associated a hazard causedby a product falling and breaking.

According to some embodiments, the detected condition may be associatedwith a queue of customers in the retail store. For example, a customermay share digital content associated with a long delay in a queue of theretail store, e.g., to request that another queue be opened. Ondetecting the condition, an alert may be issued to open a new checkoutor service counter.

According to some embodiments, the detected condition may be associatedwith a promotion for a product in the retail store. For example, acustomer may share digital content associated with a promotion of aproduct, such as availability of a new, scarce, or seasonal product (orlack thereof), availability of freshly baked goods from the bakery, or apromotional price for a product.

Some embodiments may involve automatically initiating at least oneremedial action to address the detected condition. The term“automatically” may refer to directly, spontaneously, or consequent to,e.g., without intervention or necessitating an action by an externalagent, such as a human operator. Thus, the remedial action may beinitiated automatically, e.g., directly and consequent to detecting thecondition, without intervention by an external agent and/or component.The term “remedial action” may include issuing an alert, sending anotification, sounding an alarm, invoking a robot, or performing anyother action to address the detected condition. The remedial action maybe implemented electronically as a control signal targeting a device inthe retail store, such as a robot, drone, cash register, sprinkler orfan system, or any other electronically controlled device in the retailstore, as a notification, e.g., to an emergency service, a supplier of aproduct, as an announcement to alert or inform the customers, as a queryto a database, e.g., an inventory database, or any other technique forinitiating the remedial action. For example, if the detected conditionis a shortage of a product, the remedial action may notify an employeeof the retail store to restock a shelf.

FIG. 24B illustrates an exemplary implementation for automaticallyinitiating at least one remedial action to address the detectedcondition, consistent with disclosed embodiments. On detecting condition2402 in association with the cluster constituted by posts 2452, 2454,and 2458 of FIG. 25 , processing unit 2404 may automatically invoke acleaning robot 2462 to remedy the hazard posed by condition 2402.

According to some embodiments, the at least one remedial action mayselected based on sales data associated with the particular product andthe retail store. For example, sales data for a particular product mayindicate supply and/or demand for the product, consumption patterns forthe product (e.g., seasonal, time of day, quantities, customer reactionsto promotional discounts). Thus, when determining a remedial action forthe condition, sales data may be used to tailor the remedial action forthe specific circumstances of the condition. Referring to FIG. 24B,after invoking cleaning robot 2462 to address condition 2402, processingunit 2404 may automatically place an order to restock a productassociated with condition 2402 based on a demand expectation learnedfrom sales data associated with the product.

According to some embodiments, the at least one remedial action mayinclude a prompt to restock a product display shelf in the retail store.For example, the condition may be associated with the inventory for theparticular product (e.g., a request to restock a shelf with theparticular product). The retail store may perform the remedial action byrestocking the shelf (e.g., using an automated robot). Referring to FIG.24B, processing unit 2404 may automatically place an order to restockthe product associated with condition 2402.

According to some embodiments, the at least one remedial action mayinclude a prompt to increase a number of available checkout locations.For example, the condition may be associated with a long checkout queue,e.g., after customers post complaints on social media. To remedy thecondition, a prompt may be issued to open another checkout counter, or anew checkout counter may be opened automatically (e.g., electronically).

According to some embodiments, the at least one remedial action mayinclude a prompt to clean a spill in the retail store. For example, thecondition may be associated with a spill in the retail store, and theremedial action may prompt a crew or robot to clean the spill. Referringto FIG. 24B, processing unit 2404 may prompt robot 2462 to clean aspill, e.g., to address condition 2402.

According to some embodiments, the at least one remedial action mayinclude a prompt to adjust a price associated with at least one productin the retail store. For example, the condition may be associated withan incorrectly priced product in the retail store. To remedy thecondition, a prompt may be issued to correct the price, e.g., by addingcorrect labels to the product.

FIG. 26 is a flowchart of an example process 2600 for associatingdigital activities with positions in retail stores, consistent with someembodiments of the present disclosure. Process 2600 may be performed byat least one processor or at least one processing unit, such asprocessing unit 130 or processing device 202. It is to be understoodthat throughout the present disclosure, the term “processor” is used asa shorthand for “at least one processor.” In other words, a processormay include one or more structures that perform logic operations whethersuch structures are collocated, connected, or dispersed. In someembodiments, a non-transitory computer readable medium may containinstructions that when executed by a processor cause the processor toperform process 2600. Further, process 2600 is not necessarily limitedto the steps shown in FIG. 26 , and any steps or processes of thevarious embodiments described throughout the present disclosure may alsobe included in process 2600.

In step 2602, process 2600 detects a plurality of online digitalactivities including subject matter related to a retail store, whereinthe plurality of online digital activities originate from customers inthe retail store. Referring to FIG. 24A, processing unit 2404 may detecta plurality of online digital activities (e.g., posts 2452, 2454, and2458 of FIG. 25 ) including subject matter related to a retail store,such as an identifying hashtag and image content capture from inside theretail store. Online digital activities 2452, 2454, and 2458 mayoriginate from customers 2422, 2424, and 2426 in the retail store.

In step 2604, process 2600 receives position information indicative ofpositions in the retail store from which each of the detected pluralityof online digital activities originated. Referring to FIG. 24A,processing unit 2404 may receive position information (e.g., from one ormore of sensor 2406 and mobile devices 2428, 2430, and 2432 viacommunications network 2408). The position information may be indicativeof positions of customers 2422, 2424, and 2426 in the retail store fromwhich each of posts 2452, 2454, and 2458 of FIG. 25 originated.

In step 2606, process 2600 clusters the detected plurality of onlinedigital activities based on the received position information to provideone or more clusters of detected online digital activities, wherein eachof the one or more clusters is associated with a different region of theretail store. Referring to FIG. 24A, processing unit 2404 may clusterposts 2452, 2454, and 2458 in a first cluster, and posts 2450 and 2456in a second cluster, based on the received position information, therebyproviding one or more clusters of detected online digital activities.

In step 2608, process 2600 uses an image acquisition device to captureone or more images representative of a particular region of the retailstore associated with a particular cluster of detected online digitalactivities. Referring to FIG. 24A, sensor 2406 may include an imageacquisition device to capture one or more images of the region in theretail store where customers 2422, 2424, and 2426 are situated, andassociated with the first cluster of detected online digital activities(e.g., 2452, 2454, and 2458 of FIG. 25 ).

In step 2610, process 2600 analyze the one or more captured images todetect a condition of the retail store associated with the particularcluster of detected online digital activities. Referring to FIG. 24A,processing unit 2404 may analyze one or more images received from sensor2406 via communications network 2408 to detect condition 2402 in theretail store associated with the cluster consisting of posts 2452, 2454,and 2458 of FIG. 25 .

In step 2612, process 2600 automatically initiates at least one remedialaction to address the detected condition. Referring to FIG. 24B,processing unit 2404 may automatically invoke cleaning robot 2462 toaddress condition 2462.

As described herein, the disclosed embodiments may provide variousinterfaces to allow a user to view products displayed in a retail store.In some applications, users may be interested in additional informationthan what can be gleaned from images of products on store shelves. Forexample, users may wish to track conditions of products on display in aretail store, such as quantities available, a time remaining until anexpiration date, or various other information. This can be useful forretailers wishing to maintain product displays within intendedparameters, customers wishing to acquire information on products forpurchase, or various other users as described herein. These and othertypes of information may not be readily apparent from captured images orlive video feeds.

To provide an improved interface for viewing products, the disclosedembodiments may include a hybrid visualization of retail shelves. Thishybrid visualization may include generating a synthetic imagerepresentative of one or more retail shelves to provide to a user. Thesynthetic image may include a computer-generated representation ofproducts on a shelf together with supplementary information, such asproduct quantity, placement (e.g., physical location in the retailstore), type, price, condition, orientation, etc. In response to inputfrom the user, an actual image associated with a selected region of thesynthetic image (e.g., a specific shelf, a specific productrepresentation, etc.) may be provided to the user. The actual image maybe provided in place of the synthetic image on a user interface or maybe superimposed on a portion of the synthetic image. Accordingly, thedisclosed embodiments provide, among other advantages, improvedefficiency, convenience, and functionality over existing electronicretail environment visualization systems.

FIG. 27A illustrates an example hybrid visualization interface 2700 thatmay be displayed to a user, consistent with the present disclosure.Hybrid visualization interface 2700 may be an interface configured todisplay a combination of product images as well as synthetic imagerepresentations of the products. Hybrid visualization interface 2700 maybe displayed on one or more devices to user 120, which may include acustomer, a store associate, a data collection agent, a product or brandrepresentative, or various other individuals associated with a retailenvironment. Accordingly, hybrid visualization interface 2700 may bedisplayed on one or more of devices 145A, 145B, 145C, and 145D. In someembodiments, hybrid visualization interface 2700 may be a standaloneuser interface presented to a user. Alternatively or additionally, itmay be embedded or otherwise incorporated into another user interface,such as GUI 1130 or GUI 1140, described in further detail above.Accordingly, hybrid visualization interface 2700 may be used incombination with various other display elements or features describedthroughout the present disclosure.

Hybrid visualization interface 2700 may include a synthetic visualrepresentation of one or more products on a retail shelf. For example,this may include synthetic visual representation 2730 and syntheticvisual representation 2740, as shown in FIG. 27A. Synthetic visualrepresentation 2730 and synthetic visual representation 2740 may each beconsidered a synthetic visual representation of a respective product ormay collectively be referred to as a synthetic visual representation ofmultiple products. As used herein, a synthetic visual representation mayinclude any form of computer-generated graphical element representing aproduct. For example, a synthetic visual representation may include anicon, a card (i.e., a region of associated graphical elements), a pane,a button, or various elements that may be used to represent a product.In the example, shown in FIG. 27A, synthetic visual representation 2730may represent a first milk product and synthetic visual representation2740 may represent second milk product on a retail display. In thisexample, synthetic visual representations 2730 and 2740 may be groupedbased on their location within a store. For example, synthetic visualrepresentations 2730 and 2740 may represent milk products displayed onshelf 3 of aisle 16 in a retail environment. Hybrid visualizationinterface 2700 may include various other synthetic visualrepresentations of products from other shelves, such as shelf 2 of aisle16, as indicated in FIG. 27A. Hybrid visualization interface 2700 mayinclude one or more navigation elements, such as scrollbar element 2720,which may allow a user to navigate through multiple synthetic visualrepresentations within hybrid visualization interface 2700.

While hybrid visualization interface 2700 displays the arrangement ofsynthetic visual representations of products by shelf by way of oneexample, various other arrangements may be used to present syntheticvisual representations of products. For example, this may includearranging the synthetic visual representations by popularity, proximityto a user, quantity available, price, location within a retail store,product name (e.g., alphabetically, etc.), favorite items selected by auser, products in a user's virtual shopping cart, or any otherattributes that may be associated with a product. In some embodiments,the arrangement of synthetic visual representations within hybridvisualization interface 2700 may be configurable by a user. For example,hybrid visualization interface 2700 may include one or more interactiveelements allowing user 120 to change the arrangement of synthetic visualrepresentations according to the various example arrangements describedabove, or any other arrangements. In some embodiments, hybridvisualization interface 2700 may be filterable by a user. For example,hybrid visualization interface 2700 may allow a user to filter productswithin particular price ranges, products within a certain time period ofan expiration date, products having a quantity below a specifiedthreshold, or any other properties associated with a product.

In some embodiments, a synthetic visual representation may include agraphical representation of the product. For example, this may includean image 2714, as shown in FIG. 27A. In some embodiments, image 2714 maybe a generic image of the product, such as an image of the productprovided by a manufacturer, a stock image, or the like. In someembodiments, image 2714 may be extracted from an image captured withinthe retail environment. For example, image 2714 may be an actual imageof the product captured previously using a camera 125, which may be usedto represent the product within synthetic visual representation 2730.Alternatively or additionally, image 2714 may be a vector graphic, anicon, or any other form of graphical element.

The synthetic visual representation of a product may further includeinformation about a represented product. Synthetic visual representation2730 includes various examples of information about a product that maybe displayed. In some embodiments, this may include product availabilitydata, which may include any form of data or information indicating anavailability of a product. In some embodiments, the product availabilitydata may include a binary indication of whether a product is available(e.g., in stock, available from a manufacturer, available from otherretail stores, etc.) or not available (out of stock, out of production,etc.). In some embodiments, the product availability data may include aquantity of the product, which may be a quantity placed on a retailshelf, an overall inventory of the product, or any other quantity thatmay be relevant to user 120. In some embodiments, the productavailability data may include other information, such as a product type,a brand, a manufacturer or other product attributes. For example,synthetic visual representation 2730 may include a quantity 2712 of theproduct represented by synthetic visual representation 2730. In thisexample, quantity 2712 is presented as a number of products on theshelf. In some embodiments, the quantity may be represented as an iconor other graphical element to show the number of products on a shelf, asdescribed further below.

As shown in FIG. 27A, synthetic visual representation 2730 may include aprice 2702. In some embodiments, this may include a price specified by amanufacturer, retailer, or other entity associated with product orretail store. Accordingly, price 2702 may be obtained from a database,an online data source, or other data source. As another example, price2702 may be extracted from an image of the product in the retailenvironment. For example, the price may be extracted from an image of aprice tag through OCR or similar techniques. Additional detailsregarding the source of various information presented in syntheticvisual representation 2730 are provide below. In some embodiments,synthetic visual representation 2730 may include a product type 2704.This may include a product category or any other classification of aproduct. In this example, because the product represented by syntheticvisual representation 2730 is a milk product, product type 2704 mayindicate the product is a dairy product, as shown.

Synthetic visual representation 2730 may further include productcondition information 2706. As used herein, product conditioninformation may refer to any information indicating a current state orcondition of one or more products. In the example shown in FIG. 27A,product condition information 2706 may include a temperature of theproduct, which may be obtained from one or more sensors positionedwithin the retail store. Various other types of sensor data may bedisplayed as part of condition information 2706, such as humidity,pressure, weight, lighting, or the like. In some embodiments, thecondition information may include a physical condition of a product. Forexample, this may include an indication whether one or more items of theproduct have been damaged, are leaking, etc. In some embodiments, theproduct condition information may be a time associated with the product.For example, this may be a duration the product has been on the shelf, atime remaining until a specified expiration or sell-by date, a timeuntil a next shipment of the product arrives, or any other time-relatedvalue. In some embodiments, synthetic visual representation 2730 mayrepresent more than one instance of a product on a shelf. Accordingly,the product condition information (or any other type of informationdisplayed on synthetic visual representation 2730) may be an average, aminimum, a maximum, or any other statistical value representing multipleinstances of a product. Alternatively or additionally, multiple valuesmay be presented, each corresponding to a different instance of theproduct. For example, product condition information 2706 shown insynthetic visual representation 2730 may represent an average conditionfor all products, and selecting product condition information 2706 maycause a pop-up window or other element to be displayed, which listsproduct condition information for each instance of the product.

As further shown in FIG. 27A, synthetic visual representation 2730 mayinclude a product orientation information 2708, which may indicate thedirection a product is facing on a shelf. This may be determined basedon an image of the product on the shelf or various other data describedherein. In this example, product orientation information 2708 may be anicon indicating the approximate angle of the product on the shelf (e.g.,from a top-down view). Product orientation information 2708 may bepresented in various other forms, such as a text-based indicator (e.g.,“Forward,” “Side,” “Askew,” “Upside Down,” etc.), a binary indicator ofwhether the orientation is correct or not (e.g., according to aplanogram, etc.), or the like.

In some embodiments, synthetic visual representation 2730 may includeproduct arrangement information 2710, which may be any form ofinformation indicating how the product is arranged on the shelf. Forexample, this may include an intended or planned arrangement of theproduct or an actual product arrangement on the shelf. In someembodiments, product arrangement information 2710 may be an icon-basedrepresentation of the product arrangement. In this example, productarrangement information 2710 may indicate an arrangement of the productfrom a view facing the shelf, which may indicate that the products arearranged side-by-side. In some embodiments, product arrangementinformation 2710 may represent a top-down view, or various other ways ofrepresenting the arrangement of products. Product arrangementinformation 2710 may be presented in various other forms, such as atext-based indicator (e.g., “side-by-side,” “stacked,” etc.), dimensioninformation (e.g., indicating width, depth, and height for the number ofproducts) a binary indicator of whether the arrangement is correct ornot (e.g., according to a planogram, etc.), a graphical indication(e.g., as shown in FIG. 27A) or the like. In some embodiments, theproduct arrangement information 2710 may be a more detailed view of howproducts are arranged on the shelf. For example, product arrangementinformation 2710 may be a graphical representation showing the possiblepositions of products on the shelf (similar to the arrangements shown inFIG. 9 ) with indicators to show which positions currently include aninstance of the product. This view may be displayed within syntheticvisual representation 2730 (e.g., in place of product arrangementinformation 2710) or may be displayed as a pop-up window or otherelement based on interaction with another element (e.g., based onselection of product arrangement information 2710 or quantity 2712).

In some embodiments, synthetic visual representation 2730 may furtherinclude product facing information, which may indicate the number ofitems of the product on the shelf that are turned out toward a customer.Product facing information may be presented in various forms, such as atext-based indicator (e.g., “three products wide,” etc.), a number, abinary indicator of whether the product facing is correct or not (e.g.,according to a planogram, etc.), a graphical indicator, or the like. Insome embodiments, the product facing information may be integrated withother information in synthetic visual representation 2730. In thisexample, product facing information may be represented through productarrangement information 2710.

While FIG. 27A provides an illustrative example of information that maybe displayed within a synthetic visual representation, one skilled inthe art would recognize that various other types of information. Thismay include sales data, historical data, product ratings, productreviews, recall information, product dimensions, or any other data thatmay be relevant to a user. Further, any of this data, including productavailability data, product condition information, or any otherinformation displayed in association with synthetic visualrepresentation 2730 may be obtained from a variety of sources. In someembodiments, this may include information stored in a database, such asdatabase 140. For example, product availability information may bedetermined based on sales information stored in the database, based oninformation associated with restocking events of the retail shelf storedin the database, based on return events stored in the database, based oninventory information stored in the database, and so forth.Alternatively or additionally, the information may be determined basedon image analysis of one or more images captured with a camera, such ascapturing device 125. This may include application of various imageprocessing algorithms or techniques (which may include application of aneural network) as described herein, for example, using image processingunit 130. In some embodiments, the data may be collected using a sensor,such as a pressure sensor, a weight sensor, a light sensor, a resistivesensor, a capacitive sensor, an inductive sensor, a vacuum pressuresensor, a conductive pressure sensor, an infrared sensor, aphoto-resistor sensor, a photo-transistor sensor, a photo-diodes sensor,an ultrasonic sensor, or any other sensor described herein.

Consistent with the present disclosure, hybrid visualization interface2700 may be configured to present an actual image corresponding to atleast a portion of the synthetic visual representation of the one ormore products. The actual image may include one or more image framesacquired by an image sensor, such as one or more of capturing devices125, including representations of a product. In some embodiments, theactual image may be a single frame including a representation of ashelf. This may be an image captured previously (e.g., the most recentimage captured, etc.) or may be an image captured dynamically in realtime (e.g., based on an interaction with hybrid visualization interface2700). In some embodiments, the actual image may be presented as a videoor series of images, which may be a live video feed or a previouslyrecorded video.

In some embodiments, the actual image may be displayed based on aninteraction with (e.g., a selection of) at least a portion of asynthetic visual representation. In some embodiments, this may include aselection of a particular product represented in the synthetic visualrepresentation, such as synthetic visual representation 2730. Forexample, a user may click, tap, or otherwise interact with syntheticvisual representation 2730 to cause an actual image to be displayedwithin hybrid visualization interface 2700. In some embodiments, thismay include selecting a specific element of synthetic visualrepresentation 2730, such as image 2714. As another example, hybridvisualization interface 2700 may include a dedicated element to togglebetween a synthetic visual representation view and an actual image view.For example, this may include a toggle element 2716 associated withsynthetic visual representation 2730 or a toggle element 2718 associatedwith multiple synthetic visual representations 2730 and 2740 (e.g., anentire shelf, etc.).

The actual image may be an entire image captured by capture device 125or may be a region of an image associated with a particular syntheticvisual representation or portion of a synthetic visual representation.In some embodiments, the actual image may be associated with aparticular product. FIG. 27B illustrates an example hybrid visualizationinterface 2700 displaying an actual image 2750 associated with aproduct, consistent with the present disclosure. In this example, actualimage 2750 may be an image showing the product represented by syntheticvisual representation 2730. For example, a user may select toggleelement 2716, which may cause actual image 2750 to be displayed in placeof synthetic visual representation 2730. Displaying actual image 2750may allow a user to view the current condition, arrangement,orientation, product facing, quantity, and/or other information about aproduct. For example, actual image 2750 may include a depiction of aphysical price tag 2752. Selecting toggle element 2716 again may causehybrid visualization interface 2700 to return to a view in whichsynthetic visual representation 2730 is displayed in place of actualimage 2750. While FIG. 27B illustrates actual image 2750 being displayedin place of synthetic visual representation 2730, in some embodiments,the actual image may be superimposed on a portion of synthetic visualrepresentation 2730. For example, actual image may be displayed in placeof image 2714, or otherwise superimposed on synthetic visualrepresentation 2730. As another example, actual image 2750 may bedisplayed as a pop-up window, alongside synthetic visual representation2730, in a separate interface, or through various other forms.

In the example shown in FIG. 27B, each product represented within hybridvisualization interface 2700 may be toggled between a synthetic visualrepresentation and an actual image independently. In some embodiments,the actual image may correspond to more than one product. For example, auser may select a region of a synthetic visual representationcorresponding to a specific shelf (or a portion of a shelf) and theactual image may include a representation of the shelf.

FIG. 28A illustrates an example hybrid visualization interface 2700displaying an actual image 2810 associated with a shelf, consistent withthe present disclosure. For example, a user may select toggle element2718 associated with a shelf, which may cause actual image 2810 to bedisplayed in place of the synthetic visual representation of products onthe shelf. Similar actual images may be displayed for full shelving ordisplay units (e.g., having multiple shelves), full aisles, or anyportions thereof. As with actual image 2750, in some embodiments, actualimage 2810 may be superimposed over a synthetic visual representation.

In some embodiments, hybrid visualization interface 2700 may representvarious data as a progression over time. For example, hybridvisualization interface 2700 may include a synthetic visualrepresentation conveying a progression of product availability data orother information over time. Actual images displayed through hybridvisualization interface 2700 may similarly convey information over time.For example, the actual image may be a series of actual images capturedat different points in time using an image sensor. These actual imagesmay show the progression of product availability over time.

FIG. 28B illustrates an example hybrid visualization interface 2700displaying a plurality of actual images 2820A, 2820B, and 2820Cassociated with a product, consistent with the present disclosure.Actual images 2820A, 2820B, and 2820C may be images captured by one ormore capture devices 125 at various points in time. Actual images 2820A,2820B, and 2820C may allow a user to view product availability or otherinformation over time. In some embodiments, hybrid visualizationinterface 2700 may also display information associated with each imageover time. For example, this may include displaying product availabilitydata (e.g., quantities, etc.) or various other information describedabove at a time each image was captured. In the example shown in FIG.28B, actual images 2820A, 2820B, and 2820C may be superimposed on aportion of a synthetic visual representation, which may include atimeline 2830. Timeline 2830 may include a graphical representation ofwhen actual images 2820A, 2820B, and 2820C were captured over time,including timestamp information and quantities at each capture time.While a timeline view is provided by way of example, actual images2820A, 2820B, and 2820C may be presented in various other formats. Insome embodiments, actual images 2820A, 2820B, and 2820C may be displayedsequentially as an animation, video, or the like. As another example,actual images 2820A, 2820B, and 2820C may be presented based on aninteraction with a user. For example, actual images 2820A, 2820B, and2820C may be displayed in a single frame, similar to actual image 1750,and a user may progress through actual images 2820A, 2820B, and 2820C bymoving a mouse across the actual image portion of hybrid visualizationinterface 2700. Various other elements or interfaces may be used toprogress through the actual images. This timeline or progression viewmay be displayed based on various forms of interactions with a user. Insome embodiments, the view shown in FIG. 28B may be displayed based oninteraction with toggle element 2716 (e.g., instead of the view shown inFIG. 27B, in series with the view shown in FIG. 27B, or the like).Alternatively or additionally, a separate toggle element may be includedin synthetic visual representation 2730 to bring up the view shown inFIG. 28B. As another example, the view shown in FIG. 28B may bedisplayed based on selection of another element, such as quantity 2712.

The actual images included in the series of images may be selected invarious ways and may cover various time periods. For example, actualimages 2820A, 2820B, and 2820C may cover a default period of theprevious 5 minutes, 8 hours, 24 hours, 1 week, 1 month, or any othersuitable range. In some embodiments, the period covered by actual images2820A, 2820B, and 2820C may be adjusted by a user, either dynamicallythrough hybrid visualization interface 2700 or through various settingsor options. In some embodiments, actual images 2820A, 2820B, and 2820Cmay represent all images captured during the time period. Alternativelyor additionally, hybrid visualization interface 2700 may display asubset of captured images. For example, this may include actual imagesspaced apart by a specified time interval, associated with an event(e.g., a change in product availability or other data, etc.), or variousother criteria.

FIG. 29 is a flowchart of an exemplary process 2900 for visualization ofretail shelves, consistent with the present disclosure. Process 2900 maybe performed by at least one processor or at least one processing unit,such as processing unit 130 or processing device 202. It is to beunderstood that throughout the present disclosure, the term “processor”is used as a shorthand for “at least one processor.” In other words, aprocessor may include one or more structures that perform logicoperations whether such structures are collocated, connected, ordispersed. In some embodiments, a non-transitory computer readablemedium may contain instructions that when executed by a processor causethe processor to perform process 2900. Further, process 2900 is notnecessarily limited to the steps shown in FIG. 29 , and any steps orprocesses of the various embodiments described throughout the presentdisclosure may also be included in process 2900.

In step 2910, process 2900 includes determining product availabilitydata indicative of an availability of one or more products on at leastone retail shelf in a retail store. The product availability may bedetermined in various ways. For example, at least a portion of theproduct availability data may be determined based on image analysis ofone or more images captured with a camera, such as images captured usingcapture device 125. This may include application of a visual productrecognition model, an image processing algorithm, a machine learningmodel (e.g., a trained neural network, etc.). In one example, the one ormore images may include a representation of at least a portion of the atleast one retail shelf in the retail store, may capture products placedon the at least one retail shelf in, and so forth. In some examples, amachine learning model (e.g., such as a generative model, a generativeadversarial model, etc.) may be trained using training examples togenerate availability data of products on retail shelfs from imagesand/or videos. An example of such training example may include a sampleimage and/or a sample video of a sample retail shelf, together with adesired determination of availability data of one or more sampleproducts on the sample retail shelf. The trained machine learning modelmay be used to analyze the one or more images captured with the cameraand determine the product availability data. Alternatively oradditionally, at least a portion of the product availability data may bedetermined based on information stored in a database. For example, thismay include data stored in database 140, as described above.

In step 2920, process 2900 includes using the product availability datato generate a synthetic visual representation of the one or moreproducts on the at least one retail shelf. For example, this may includegenerating one or more of synthetic visual representation 2730 andsynthetic visual representation 2740. As described above, this mayinclude generating synthetic visual representation 2730 to includequantity 2712 or various other forms of product availability data. Asdescribed above, the synthetic visual representation may convey otherforms of information associated with a product. For example, process2900 may further include determining one or more of product-facinginformation, product condition information, product arrangementinformation, product orientation information, a quantity, a price, aproduct type, or any other information associated with a product, whichmay be conveyed via the synthetic visual representation. Additionalexamples of information that may be conveyed via the synthetic visualrepresentation are provided above. In some embodiments, the syntheticvisual representation may be generated at least partially using a neuralnetwork. For example, a neural network may be trained to extractinformation about a product, such as a quantity, a product arrangement,or the like, which may be conveyed via the synthetic visualrepresentation. In one example, a machine learning model (e.g., such asa generative model, a generative adversarial model, etc.) may be trainedusing training examples to generate synthetic visual representation ofthe products on the retail shelves from product availability data. Anexample of such training examples may include a sample productavailability data, together with a desired synthetic visualrepresentation corresponding to the sample product availability data.The trained machine learning model may be used to analyze the productavailability data and generate the synthetic visual representation ofthe one or more products on the at least one retail shelf.

In step 2930, process 2900 includes presenting the synthetic visualrepresentation on a display. For example, this may include presentingthe synthetic visual representation via a display of an output device145 (including devices 145A, 145B, 145C, and 145D). In some embodiments,the synthetic visual representation may be presented to the user via auser interface shown on the display, such as hybrid visualizationinterface 2700, as shown in FIG. 27A.

In step 2940, process 2900 includes receiving from a user an inputindicative of a selection of at least a portion of the synthetic visualrepresentation. In some embodiments, the input may include a selectionof a region of the synthetic visual representation, which may correspondto the one or more products. For example, this may include a selectionof synthetic visual representation 2730 or toggle element 2716, asdescribed above. In some embodiments, the selected region may correspondto the at least one retail shelf, such as an area including bothsynthetic visual representation 2730 and synthetic visual representation2740, toggle element 2718, or the like.

In step 2950, process 2900 includes presenting on the display an actualimage corresponding to the selected at least a portion of the syntheticvisual representation, which may be presented in response to receivingthe input in step 2940. The actual image may have been acquired with animage sensor, such as image sensor 310. For example, step 2950 mayinclude displaying one or more of actual images 2750 or 2810, asdescribed above. In some embodiments, the actual image may be presentedon the display in place of the synthetic visual representation, as shownin FIG. 27B. Alternatively or additionally, the actual image may bepresented as superimposed on a portion of the synthetic visualrepresentation. For example, an image matting algorithm may be used tostitch the actual image (or portions thereof) into the synthetic visualrepresentation. In some examples, the actual image may be selected of aplurality of alternative actual images corresponding to the at least aportion of the synthetic visual representation. In one example, theselection of the actual image may be based on a mode selected by theuser. For example, in a ‘current status’ mode, the most recent actualimage corresponding to the at least a portion of the synthetic visualrepresentation may be selected of the plurality of alternative actualimages, while in a ‘worst observed status’, the actual imagecorresponding worst observed status of the plurality of alternativeactual images may be selected. In another example, an image thatreceives a highest (or lowest) score of the plurality of alternativeactual images may be selected. Some non-limiting examples of such scoremay include sharpness score, a score associated with image quality, ascore associated with planogram compliance, a score associated withactual sales data, and so forth. In some embodiments, the syntheticvisual representation may convey a progression over time of the productavailability data and the actual image may include a plurality of actualimages corresponding to the selected at least a portion of the syntheticvisual representation captured at different points in time using theimage sensor. For example, this may include displaying actual images2820A, 2820B, and 2820C, as described above. Accordingly, the pluralityof actual images may show the progression over time of the productavailability data.

As described herein, one or more image capture devices may be placed ina retail store to determine information associated with productsdisplayed in the retail store. As store operators and customersincreasingly rely on data acquired from these image capture devices, itmay be beneficial to ensure the devices are installed correctly and areproperly maintained. For example, some retail stores may provide africtionless shopping experience in which the checkout process acustomer must complete to make a purchase is expedited or eveneliminated by tracking a customer's product selections throughout theirvisit. Accordingly, it may be beneficial to ensure that image capturedevices are placed to provide sufficient coverage of the retail storeand that the image capture devices are working properly.

This installation and maintenance of image capture devices may bedifficult for store associates. When initially placing image capturedevices, it may be difficult to estimate what field of view the imagecapture device will cover. Accordingly, an associate or other installermay perform an iterative process of placing one camera, examining animage captured by the camera, and adjusting a camera position, which maybe repeated multiple times. This may also require several iterations toensure that proper coverage is achieved when using multiple cameras(e.g., with minimal overlap, with no blind spots, etc.). Further, usingconventional techniques, it may be difficult for an associate or otherindividual to easily ascertain the status or other operating parametersof image capture devices after they are installed, especially fromwithin an aisle or other area where the image capture devices areplaced.

To address these and other issues, the disclosed embodiments leveragethe use of extended reality technology to assist with the placement andmaintenance of cameras in a retail space. The disclosed system cantransmit camera status information to a user's extended reality device.For example, this camera status information may include whether one ormore cameras are operational, camera locations, a field of view of oneor more cameras, a camera pointing direction, a frame rate, aresolution, or the like. This information may be overlaid on the retailstore environment through the extended reality device to assist anassociate with new camera placements, camera upgrades, cameraadjustments, or various other camera maintenance or installationoperations. Accordingly, the disclosed embodiments provide, among otheradvantages, improved efficiency, convenience, and functionality overexisting electronic retail environment maintenance systems.

FIG. 30A illustrates an example view 3000 of a retail store that may beenhanced by an extended reality device 3010, consistent with thedisclosed embodiments. View 3000 may represent a view from the point ofview of an associate using extended reality device 3010. View 3000 maybe enhanced through extended reality device 3010 to overlay a visualelement, such as status indicator 3030 relative to the physical locationof cameras in the retail store, as described further below.

As used herein, an extended reality device may include any deviceenabling the combined visualization of real and virtual elements. Forexample, extended reality may include one or more of augmented reality,mixed reality, and virtual reality, or any similar form of mediated orhybrid reality. In some embodiments, extended reality device 3010 mayinclude a see-through display in which status information or othervisual elements are projected on or otherwise displayed on a see-throughmedium. This may include a handheld device including see-throughdisplay, a head-mounted device, or any other form of optical see-throughdisplay. For example, extended reality device 3010 may include a pair ofsmart glasses, an add-on device for conventional glasses, a pair ofsmart contact lenses, a helmet, a heads up display (HUD), or the like.Accordingly, view 3000 may represent the view of an associate through asee-through display and status indicator 3030 and other information maybe projected onto or otherwise presented through the see-throughdisplay. As another example, extended reality device 3010 may include ahandheld device including a display screen and a camera. The handhelddevice may be configured to display images captured using the camera inreal time (or near real time) and overlay additional visual elements onthe captured images. This form of device may also be referred to as avideo see-through display device. For example, this may include a tabletor mobile device, which a store associate may carry when installing ormaintaining cameras. Accordingly, view 3000 may represent a videodisplay screen displaying a captured image frame with status indicator3030 and other information overlaid on the image. In some embodiments,extended reality device 3010 may correspond to output device 145Cdescribed in further detail above. Accordingly, any details orembodiments described above with respect to output device 145C mayequally apply to extended reality device 3010.

The disclosed systems may receive, access, or otherwise acquire cameralocation information associated with a plurality of cameras deployed ina retail store. For example, camera location information may be storedin database 140 and may be accessed by server 135. Alternatively oradditionally, camera location information may be received throughcommunications network 150 from another device or entity within system100, such as a device associated with retail store 105. The cameralocation information may be any information indicating the physicallocation of one or more cameras in the retail store. For example, thismay include horizontal position information (e.g., relative to floorplanor map of a retail store), height information, orientation information,shape information, or any other information that may define a physicalposition of a camera. In some embodiments, the location information maybe defined relative to a coordinate system of a retail environment. Asanother example, the location information may be presented in a formatindicating an aisle number, shelf number, lateral shelf position,pointing direction, or other similar information. Various other forms oflocation information, including GPS data, or other information may beused.

The disclosed systems may also receive or access camera statusindicators for one or more cameras. In some embodiments, the camerastatus information may be stored in database 140 and may be accessed byserver 135. Alternatively or additionally, the camera status informationmay be received through communications network 150 from another deviceor entity within system 100. For example, server 135 may communicate(either directly or indirectly) with one or more capturing devices 125.The camera status indicators may include a wide variety of informationassociated with a camera. This may include power information (e.g.,on/off status, power consumption, battery level, battery health, etc.),a camera mode (e.g., actively capturing, standby mode, etc.), camera orsensor properties (e.g., frame rate, resolution, number of imagescaptured, image file size, sensor type, zoom level, aperture, focallength, angle of view, etc.), field of view information, image data,orientation information, or any other information associated with acamera. This status information may include current information, maximumor minimum information, average values, or any other form of statisticalinformation associated with a camera. In some embodiments, the statusinformation (or location information) may include 3D shape datarepresenting a physical shape and/or appearance of the camera, which maybe used to represent the camera in an extended reality view.

The camera location information and camera status indicators may be usedto generate a digital signal, which may be provided to extended realitydevice 3010 either directly or through various intermediate devices. Thedigital signal may cause extended reality device 3010 to display avisual representation of a status indicator for one or more cameras.Accordingly, the digital signal may include instructions to extendedreality device 3010 to display information associated with one or morecameras. The digital signal may further include information from which avisual representation may be derived. For example, the digital signalmay include the camera location information and camera status indicatorsfor one or more cameras in a retail store associated with extendedreality device 3010. Extended reality device 3010 may be configured toprocess this data to generate a visual representation of the statusindicators to display to the user. Alternatively or additionally, someor all of the processing of the camera location information and statusindicator information may be processed by server 135. For example, thismay include extracting status information, correlating it to cameralocation information, and presenting the information in a format that isrecognized by extended reality device 3010.

Consistent with the disclosed embodiments, the visual representation ofthe status indicators may be displayed relative to the actual positionsof camera in the retail store. Accordingly, extended reality device 3010may be configured to monitor the area of a retail store included in view3000 and adaptively display the visual representation of statusindicators relative to the position of the cameras in view 3000. As oneskilled in the art would recognize, this may include analyzingaccelerometer data, beacon device data, orientation sensor data, imagedata, or various other forms of data or combinations thereof todetermine a position and orientation of view 3000.

As shown in FIG. 30A, view 3000 may include various cameras 3020, 3022,and 3024 positioned within the retail store. Cameras 3020, 3022, and3024 may be configured to capture images of products displayed withinthe retail store, as described herein. For example, cameras 3020, 3022,and 3024 may correspond to capturing devices 125. Server 135 may accesscamera location information and status indicators associated withcameras 3020, 3022, and 3024, and generate a digital signal as describedabove. Extended reality device 3010 may receive the digital signal anddisplay status indicator 3030. As shown in FIG. 30A, status indicator3030 may be a graphical element including various status informationassociated with camera 3020. Status indictor 3030 may be presented invarious other forms, as described further below. Status indicator 3030may be presented at a location within view 3030 corresponding to camera3020, as shown. This may help a user of extended reality device 3010locate camera 3020 within the retail store and quickly access relevantinformation about camera 3020.

As described above, the status information may include a variety ofinformation about camera 3020, any of which may be included in statusindicator 3030. For example, as shown in FIG. 30A, status indicator 3030may include an identifier 3032 of camera 3020. This may include a name,a number, an alphanumeric string, or any other identifier assigned tocamera 3020, which may help a user of extended reality device 3010identify which cameras are placed in which locations. In someembodiments, status indicator 3030 may include a power status 3034 ofcamera 3020. In this example, power status 3034 may indicate that camera3020 is powered on. Power status 3034 may indicate various other powerinformation, such as an offline status, a standby status, a power-savingstatus, or the like. Status indicator 3030 may further include a batterylevel 3036, which may indicate a remaining battery level of camera 3020.Various other forms of power-related information may also be conveyed,including a battery health status, a charging status, a time since lastcharge, power usage information (e.g., voltage, current, amperage,etc.), or the like. As another example, status indicator 3030 mayinclude data connectivity information, such as whether a communicationcan be established with camera 3020, a data transfer rate, a signalstrength, a signal type (e.g., Wi-Fi™, Zigbee™, Z-wave™, wired, etc.),or various other forms of connectivity information.

In some embodiments, status indicator 3030 may include operatingparameters associated with camera 3020. This may include current ormeasured values, or nominal or specified operating parameters, such asmaximum or minimum operating parameters, target operating parameters, orthe like. In some embodiments, status indicator 3030 may include acamera frame rate identifier 3042, which may convey a current framerate, a nominal frame rate, or the like. As another example, statusindicator 3030 may include a camera resolution identifier 3044, whichmay indicate a resolution of images captured using camera 3020. As withother operating values, this may include a current resolution setting, amaximum resolution (e.g., of an image sensor), or the like. In someembodiments, status indicator 3030 may include a camera angulardirection identifier 3046, which may indicate a field of view angle ofcamera 3020 as well as an indication of the direction of camera 3020. Insome embodiments, extended reality device 3010 may display a directionindicator 3048 in the form of an arrow, a set of simulated axes, anangular identifier, or other directional indicator. Direction indicator3048 which may be overlaid on a lens or other portion of camera 3020indicating the direction camera 3020 is facing.

The information conveyed in status indicator 3030 is provided by way ofexample, and various other information associated with a camera may alsobe displayed. For example, this may include a camera installation date,a date maintenance was performed on the camera, a make or model number,a camera type, an indication of one or more events detected using thecamera, or any other information that may be useful to a store associateor other user of extended reality device 3010. Further, while statusindicator 3030 is shown relative to camera 3020 only in FIG. 30A forpurposes of clarity, it is to be understood that visual representationsof status indicators may also be displayed for other cameras, includingcameras 3022 and 3024. In some embodiments, visual representations ofstatus indicators may be displayed for a subset of cameras in the retailstore. For example, extended reality device 3010 may only display visualrepresentations of status indicators for cameras that are currently inview 3000 (which may be determined based on analysis of images capturedby extended reality device 3010, etc.), cameras within a predetermineddistance from extended reality device 3010, cameras meeting specifiedstatus criteria (e.g., cameras needing maintenance, cameras that arepowered on or off, etc.), or various other criteria. In someembodiments, cameras may be associated with certain products or productcategories based on which products are included in a field of view ofthe camera. Accordingly, extended reality device 3010 may allow a userto view visual representations of status indicators associated withspecified products or product types. Various other forms of filters maybe applied, such as cameras associated with a particular aisle, aparticular supplier, a particular region of the retail store, or thelike.

Various other forms of information may be displayed through extendedreality device 3010. FIGS. 30B-30D illustrate various other exampleinformation that may be displayed within view 3000 using extendedreality device 3010, consistent with the disclosed embodiments. In someembodiments, this may include virtual position indicators showing theactual positions of the cameras. This may allow a user of extendedreality device 3010 to quickly and easily locate cameras within theretail environment. For example, as shown in FIG. 30B, a bounding box3050 may be displayed around camera 3020 so that it may be easilyidentified by a user of extended reality device 3010. While a box isshown in FIG. 30B by way of example, this may include a triangle,circle, oval, or other form of geometric shape. In some embodiments, theposition indicator may be a bounding shape configured to trace anoutline of camera 3020. This shape may be determined based on analysisof an image captured using extended reality device 3010, 3D shape datareceived as part of camera location information or status information,or any other data that may indicate the portion of view 3030 thatincludes camera 3020.

As another example, the position indicator may include a shape 3060overlaid on camera 3020, as shown in FIG. 30C. Shape 3060 may include acolor to highlight the position of camera 3020 in view 3000. In someembodiments, the color may reflect status information. For example, agreen highlighting may indicate camera 3020 is powered on and operatingcorrectly, a yellow highlighting may indicate that camera 3020 needsadjustment or other maintenance, and a red highlighting may indicatethat camera 3020 is offline or not operating correctly. Various otherinformation may be conveyed through a color of shape 3060. As anotherexample, the position indicators may include a pointer, an icon, oranother graphic displayed relative to camera 3020 (e.g., overlaid on thecamera, above the camera, below the camera, to the side of the camera,etc.). In some embodiments, the position indicators may be expanded todisplay additional status information, similar to status indicator 3030shown in FIG. 30A. For example, this may include expanding positionindicators for cameras within a predetermined range of extended realitydevice 3010, cameras requiring attention from a user, based on theposition of the cameras in view 3000 (e.g., to avoid overlapping statusindicators, etc.), based on a product or product category included in afield of view of the camera, or the like. In some embodiments, aposition indicator may be expanded based on an input from a user ofextended reality device 3010. For example, a user may select a positionindicator through a hand gesture, a voice command, interacting with aphysical display, or the like, which may cause an expanded view toappear.

In some embodiments, the visual representation of a status indicator fora camera may include an image captured by a camera. For example, asshown in FIG. 30D, extended reality device 3010 may display an image3070 captured using camera 3020. This may allow a user of extendedreality device 3010 to assess a field of view of camera 3020, which mayassist with aligning camera 3020, verifying camera coverage areas,assessing image quality, or the like. In some embodiments, image 3070may be a representative image previously captured using camera 3020. Forexample, this may include an image captured when camera 3020 wasinstalled, the last image captured using camera 3020, an image capturedbased on a use of extended reality device 3010 (e.g., when camera 3020comes into view 3000, an input from a user, when an expanded view isdisplayed, etc.), or any other images captured using camera 3020.Alternatively or additionally, image 3070 may be a live video feed fromcamera 3020.

In some embodiments, the visual representation of the status indicatormay provide a visual representation of a field of view of a camera.FIGS. 31A-31D illustrate example visual representations of a field ofview of one or more cameras, consistent with the disclosed embodiments.As used herein, a visual representation of the field of view may be anyvisual element that at least partially conveys which portion of a retailstore is represented in images captured using a camera. In someembodiments, the field of view of a camera may be determined based on aknown angle of view of a camera (e.g., based on a make and model of acamera and associated technical specifications) and a location of thecamera. As another example, the field of view may be determined based onimages captured by a camera. For example, extended reality device 3010or server 135 may correlate objects detected in an image captured usingcamera 3020 with representations of objects detected in view 3000.Various other means for determining a field of view of a camera may beused.

According to some embodiments, the visual representation of a field ofview of a camera may include a two-dimensional map view of the camerafield of view relative to the retail store. For example, as shown inFIG. 31A, extended reality device 3010 may display a two-dimensional map3110 which may represent a layout of a retail store (e.g., an overheadview) in which camera 3020 is placed. Two-dimensional map 3110 may alsoinclude a location indicator 3112, which may be an icon, shape, or othervisual element overlaid on two-dimensional map 3110 to represent alocation of a field of view of camera 3020 relative to the retail store.Alternatively or additionally, the visual representation of the field ofview of a camera may include a visual indication of a physical portionof the retail store corresponding to the camera field of view. Forexample, this may include a three-dimensional shape 3114 presented in anextended reality environment allowing a user of extended reality device3010 to visualize a field of view of camera 3020. In the example shownin FIG. 31A, three-dimensional shape 3114 may be a set of edges definingthe boundaries of a field of view of camera 3020. Whilethree-dimensional shape 3114 is shown as a set of edges in FIG. 31A,three-dimensional shape 3114 may be represented in various other ways,including an opaque or semi-opaque object, a shaded region of view 3000,or the like.

In some embodiments, the visual representation of the field of view of acamera may include a visual indication in an extended realityenvironment of a portion of a surface area of a physical object in theretail store corresponding to the camera field of view. For example,extended reality device 3010 may display a visual indication identifyinga portion 3120 of a surface of a shelving 3002 included in a field ofview of camera 3020. Similarly, extended reality device 3010 may displaya portion 3122 of a surface of a shelving 3002 included in a field ofview of camera 3022. This may allow a user of extended reality device3010 to easily view which portions of shelving 3002 is within the fieldof view of a camera. Accordingly, the user of extended reality device3010 may easily identify areas where additional cameras need to beplaced, identify existing cameras need to be adjusted, upgraded, orotherwise reconfigured, or the like. Portions 3120 and 3122 may berepresented as two-dimensional shapes placed on the surface of shelving3002 or any other surface in a retail store. In FIG. 31B, portions 3120and 3122 are displayed as opaque shapes, however, they may berepresented in various other ways, including a semi-transparent shape,an outline, or the like. In some embodiments, portions 3120 and 3122 mayinclude an image captured by cameras 3020 and 3022, respectively,similar to image 3070. This may allow a user to verify the accuracy ofportions 3120 and 3122 by comparing the displayed image to thesurrounding environment.

In some embodiments, extended reality device 3010 may be display aproposed location for an installation of an additional camera. Forexample, this may include displaying a visual indicator 3124 of aproposed location for placing an additional camera in the retail store.The proposed location may be selected to ensure that proper coverage isprovided by the cameras. For example, as shown in FIG. 31B, shelving3002 may include a gap that is not covered by the cameras in the retailstore. Accordingly, visual indicator 3124 may indicate a proposedlocation that would ensure this gap is captured. As shown in FIG. 31B,visual indicator 3124 may be a pointer, an icon, or another indicatorspecifying the proposed location. In some embodiments visual indicator3124 may also include proposed technical specifications that wouldaddress the coverage gap, similar to status indicator 3030. The proposedlocation may be presented in various other ways, including a 3Drendering of the proposed camera, a bounding box or shape, a coloredhighlight, or the like. The proposed locations may be determined byserver 135 (and included in the digital signal), by extended realitydevice 3010, or by various other devices in system 100.

Consistent with some embodiments of the present disclosure, the field ofview of a camera may be determined at least in part based on a locationof a physical object in the retail store. For example, the object mayobstruct a portion of the field of view of a camera and this obstructionmay be reflected in the visual representation of the field of view ofthe camera. The location of objects may be determined in various ways.In some embodiments, the locations of objects may be stored in adatabase (e.g., database 140) or another storage location. As anotherexample, the objects may be detected in one or more images capturedusing cameras in the retail store and the location of the objects may bedetermined from the image data.

FIG. 31C illustrates an example field of view represented asthree-dimensional shape 3132, which may be affected by an object 3130 inthe retail store. In this example, object 3130 may be a promotionalsign, a caution sign, or another form of sign that may be displayed in aretail store. The location of object 3130 may be obtained based onanalysis of images captured using camera 3020, based on stored locationinformation, or the like, as described above. As shown in FIG. 31C, thefield of view of camera 3020 may be determined using the location ofobject 3130, and three-dimensional shape 3132 may indicate that at leasta portion of the field of view of camera 3020 is obstructed. The fieldof view may be displayed as a portion of a surface of an object (similarto portion 3120), or with various other visual elements. In someembodiments, extended reality device 3010 may display a recommendationto relocate the object to thereby extend the camera field of view. Forexample, this may include displaying an indicator 3134 recommendingobject 3130 be relocated. Indicator 3134 may be displayed in variousother manners, including a bounding box, a highlighting, or the like. Insome embodiments, the recommendation may be determined based on anobject type associated with the physical object. For example, this mayinclude refraining from providing recommendations to move stationaryobjects (e.g., structural supports, etc.), objects placed according tocontractual obligations (e.g., promotional displays, etc.), or variousother categories or properties of products that may affect whether theobject should be moved.

According to some embodiments, extended reality device 3010 may displaya visual indication of at least one physical portion of the retail storenot covered by the plurality of cameras. For example, this may includedisplaying portion 3140 representing a surface of shelving 3002 notcovered by cameras of the retail store. Portion 3140 may correspond to agap in coverage as illustrated in FIG. 31B and described above. Whileportion 3140 is shown as a portion of a surface in FIG. 31D, portion3140 may be presented in various other ways, such as a three-dimensionalshape, or the like. In some embodiments, extended reality device 3010may further include a visual indicator of a proposed location of acamera that would address portion 3140, such as visual indicator 3124described above. In some embodiments, the portion of the retail storerepresented by portion 3140 may represent a portion of the physicalportion of the retail store not covered by the plurality of cameras.Accordingly, portion 3140 may be selected in various ways. In someembodiments, portion 3140 may be selected based on a product type orproduct category associated with the physical portion of the retailstore. For example, this may include prioritizing some types orcategories of products over others when displaying portions not coveredby the plurality of cameras. Accordingly, portion 3140 may represent aportion of a store associated with a certain product that is not coveredby a camera. In some embodiments, extended reality device 3010 maydisplay an indication of the product time, for example through anindicator 3142.

As another example, portion 3140 may be selected based on sales dataassociated with products associated with the physical portion of theretail store. For example, the system may prioritize more popular orprofitable products over other products. Similarly, portion 3140 may beselected based on a price associated with products associated with thephysical portion of the retail store. For example, the system mayprioritize ensuring that higher-priced items are included in a field ofview of a camera relative to lower-priced items. As another example,portion 3140 may be selected based on a planogram associated with thephysical portion of the retail store. For example, the system mayhighlight portions of the retail store associated with a particularplanogram that are not covered by a camera field of view. Various othercharacteristics of the retail store or products may be used to selectthe portion of the retail store not covered by the plurality of camerasthat is represented by portion 3140.

FIG. 32 is a flowchart of an exemplary process 3200 for facilitatingcamera system inspection in an extended reality environment, consistentwith the present disclosure. Process 3200 may be performed by at leastone processor or at least one processing unit, such as processing unit130 or processing device 202. It is to be understood that throughout thepresent disclosure, the term “processor” is used as a shorthand for “atleast one processor.” In other words, a processor may include one ormore structures that perform logic operations whether such structuresare collocated, connected, or dispersed. In some embodiments, anon-transitory computer readable medium may contain instructions thatwhen executed by a processor cause the processor to perform process3200. Further, process 3200 is not necessarily limited to the stepsshown in FIG. 32 , and any steps or processes of the various embodimentsdescribed throughout the present disclosure may also be included inprocess 3200.

In step 3210, process 3200 includes receiving camera locationinformation associated with a plurality of cameras deployed in a retailstore. As described further above, the camera location information beingindicative of an actual position of each camera of the plurality ofcameras. For example, this may include receiving camera locationinformation indicating a physical location of cameras 3020, 3022, and3024. The camera location information may be retrieved from a database,received from a store system, or various other sources within system100.

In step 3220, process 3200 includes receiving, for each camera of theplurality of cameras, at least one camera status indicator. For example,this may include receiving a camera status indicator for cameras 3020,3022, and 3024. As described in further detail above, the camera statusindicator may include various types of information associated with acamera. For example, this may include power information, connectivityinformation, camera operating parameters, camera operating modes,captured image data, or any other forms of status information asdescribed above.

In step 3230, process 3200 includes using the received camera locationinformation and the received camera status indicators to generate adigital signal. In some embodiments, generating the digital signal mayinclude processing the received camera location information and thereceived camera status indicators to provide the information in a formatrecognized by extended reality device 3010, filtering relevantinformation, or the like. Alternatively or additionally, the digitalsignal may include the received camera location information and thereceived camera status indicators, which may be processed further byextended reality device 3010.

In step 3240, process 3200 includes providing the generated digitalsignal to an extended reality device to thereby cause the extendedreality device to display a visual representation of the at least onestatus indicator for each camera of the plurality of cameras relative tothe actual position of the camera in the retail store. For example, thismay include providing the generated digital signal to extended realitydevice 3010 described above. Accordingly, the extended reality devicemay include a handheld device (e.g., having a video see-through display)or a device having an optical see-through display. In some embodiments,the extended reality device may include a head mounted device, such as apair of smart glasses, a helmet, one or more smart contact lenses, orthe like.

The digital signal may cause extended reality device 3010 to displayvarious visual representations of the at least one status indicator foreach camera, as described above. In some embodiments, the visualrepresentation of the at least one status indicator may convey at leastone of a camera battery level or a camera power on status. For example,this may include displaying power status 3034 and battery level 3036, asdescribed above. As another example, the visual representation of the atleast one status indicator may convey at least one of a camera framerate identifier, a camera resolution identifier, a camera angulardirection identifier, or other camera information. For example, this mayinclude camera frame rate identifier 3042, camera resolution identifier3044, camera angular direction identifier 3046, and/or directionindicator 3048, as described above. As another example, the visualrepresentation of the at least one status indicator for a particularcamera may include a sample image captured by the particular camera,such as image 3070.

In some embodiments, providing the generated digital signal to theextended reality device may further cause the extended reality device todisplay in an extended reality environment virtual position indicatorsof the actual positions of the cameras in the retail store. For example,this may include a color highlight, such as shape 3060, a bounding box,such as bounding box 3050, a pointer, an icon, or a similar visualindicator. As another example, providing the generated digital signal tothe extended reality device may further cause the extended realitydevice to display in the extended reality environment a visualindication of a proposed location for an installation of an additionalcamera. For example, this may include displaying visual indicator 3124of a proposed location for placing an additional camera in the retailstore, as described above.

In some embodiments, the visual representation of the at least onestatus indicator may provide a visual representation of a camera fieldof view. The visual representation of the camera field of view may bedisplayed in various forms as described above. For example, the visualrepresentation of the camera field of view may include a two-dimensionalmap view of the camera field of view relative to the retail store, suchas two-dimensional map view 3110. This may also include a graphicalrepresentation of the field of view as shown in FIG. 31A. As anotherexample, the visual representation of the camera field of view mayinclude a visual indication in an extended reality environment of aphysical portion of the retail store corresponding to the camera fieldof view. For example, this may include displaying a three-dimensionalshape 3114 representing the camera field of view, as described infurther detail above. In some embodiments, the visual representation ofthe camera field of view may include a visual indication in an extendedreality environment of a portion of a surface area of a physical objectin the retail store corresponding to the camera field of view. Forexample, this may include portions 3120 and 3122, as described above,which may indicate a portion of a surface of shelving 3002 thatcorresponds to the field of view of cameras 3020 and 3022, respectively.

In some embodiments, the field of view may account for the presence ofobjects or other obstacles in the view of cameras of the retail store.Accordingly, process 3200 may further include receiving object locationinformation associated with a physical object in the retail store andusing the object location information to determine the camera field ofview. For example, this may include receiving a location of object 3130,as described above, which may be used to determine the field of viewrepresented by three-dimensional shape 3132. As described above, process3200 may further include providing, through the extended reality device,a recommendation to relocate the object to thereby extend the camerafield of view. For example, this may include displaying an indicator3134 recommending that object 3130 be relocated. According to someembodiments, the recommendation may be determined based on an objecttype associated with the physical object.

Consistent with the disclosed embodiments, providing the generateddigital signal to the extended reality device may further includecausing the extended reality device to display in an extended realityenvironment a visual indication of at least one physical portion of theretail store not covered by the plurality of cameras. The at least onephysical portion of the retail store not covered by the plurality ofcameras may be selected according to various criteria. For example, theat least one physical portion of the retail store not covered by theplurality of cameras may be selected from a physical portion of theretail store not covered by the plurality of cameras based on a producttype associated with the at least one physical portion of the retailstore (e.g., product types may be ranked by importance by a user or anautomated process, and at least one physical portion of the retail storeassociated with the highest ranking product type that is not covered bythe plurality of cameras may be selected), a product category associatedwith the at least one physical portion of the retail store (e.g.,product categories may be ranked by importance by a user or an automatedprocess, and at least one physical portion of the retail storeassociated with the highest ranking product category that is not coveredby the plurality of cameras may be selected), a planogram associatedwith the at least one physical portion of the retail store (e.g., atleast one physical portion of the retail store associated with theplanogram or with a selected portion of the planogram may be selected),a sales data associated with products associated with the at least onephysical portion of the retail store (e.g., at least one physicalportion of the retail store associated with highest sales numbers andnot covered by the plurality of cameras may be selected), a priceassociated with products associated with the at least one physicalportion of the retail store (e.g., at least one physical portion of theretail store associated with highest prices and not covered by theplurality of cameras may be selected), or various other informationassociated with a retail store. In some examples, a model of the retailstore may be analyzed, for example based on fields of view of theplurality of cameras, to determine portions of the retail store that arecovered by the plurality of cameras, and a model of the retail store maybe analyzed based on the determined portions of the retail store thatare covered by the plurality of cameras to determine the at least onephysical portion of the retail store not covered by the plurality ofcameras, for example by selecting a physical portion of the retail storethat correspond to a portion of the model that does not correspond toany of the determined portions of the retail store that are covered bythe plurality of cameras. In some examples, images captured using theplurality of cameras may be analyzed to determine portions of the retailstore that are covered by the plurality of cameras, and a model of theretail store may be analyzed based on the determined portions of theretail store that are covered by the plurality of cameras to determinethe at least one physical portion of the retail store not covered by theplurality of cameras, for example by selecting a physical portion of theretail store that correspond to a portion of the model that does notcorrespond to any of the determined portions of the retail store thatare covered by the plurality of cameras. In some examples, imagescaptured using the plurality of cameras may be received, and an imagecaptured using the extended reality device may be analyzed to identifyin the image captured using the extended reality device a portion thatis not included in the images captured using the plurality of cameras,for example by comparing the content of the image captured using theextended reality device to the content of the received images capturedusing plurality of cameras. In some examples, images captured using theplurality of cameras may be analyzed using a product recognitionalgorithm to determine product types covered by the plurality ofcameras, and the at least one physical portion of the retail store notcovered by the plurality of cameras may be selected to include an areaof the retail store corresponding to one or more product types notcovered by the plurality of cameras. In some examples, images capturedusing the plurality of cameras may be analyzed using a visualclassification algorithm to determine product categories covered by theplurality of cameras, and the at least one physical portion of theretail store not covered by the plurality of cameras may be selected toinclude an area of the retail store corresponding to one or more productcategories not covered by the plurality of cameras.

A primary hurdle a retail store experiences in managing its inventory iskeeping shelves stocked when products are in high demand. Point of saledata can assist in the management of store inventory, but additionalproblems can quickly develop as unexpected or anomalous transactionsoccur that deviate from expected sales patterns. For instance, point ofsale data for a given customer demographic may suggest that a customerof a certain age, gender, or historical shopping experience is likely topurchase a product of a particular type. However, a customer mayinstead, from time to time, go against that historical trend ordemographic pattern and purchase a product of a different type. This mayoccur for a variety of reasons. As such, a retail store may not be ableto anticipate an anomalous transaction based on previous point of saledata.

To address anomalous transactions, receipt of the point of sale data maycause image data capturing technology in the store to capture additionalinformation about the anomalous condition for analysis and to ultimatelydetermine its cause. For example, if a customer purchases an unexpectedproduct type such as white bread when previous sales data suggests thatcustomer traditionally purchases wheat bread, that sale could beconsidered an anomalous transaction. After identifying the anomaloustransaction (e.g., by point of sale data reflecting a purchase of whitebread), the disclosed systems and methods may cause cameras or othertechnology installed in the retail store to capture information that canbe used to determine the cause of the anomalous transaction. Forexample, the captured image data may show that the shelves are currentlyout of stock of wheat bread, or perhaps the quality of the wheat breadappears lacking (e.g., stock is in poor condition, past an expirationdate, etc.), or perhaps a sale on white bread caused the customer topurchase white bread instead of wheat bread. Depending on theinformation determined from the captured image data, additional actionsor notifications may be generated to rectify or correct the store'sinventory of wheat bread. This action or information may cause anythingfrom a notification to a retail store associate about the anomalouscondition that led to the anomalous transaction to an automatedresponse, such as deployment of a robot or drone to restock shelves. Thedisclosed systems and methods, described in further detail below, seeksto seamlessly and accurately manage the inventory of a retail store andaddress anomalous transactions.

In an embodiment, a non-transitory computer-readable medium may includeinstructions that when executed by at least one processor cause the atleast one processor to perform a method for triggering actions inresponse to point of sales data. The non-transitory computer-readablemedium may include a memory device of a server, laptop, personal desktopcomputer, smart phone, tablet, or similar data processing machine.Further details regarding the method are described below in connectionwith FIGS. 33, 34, and 35 .

FIG. 33 shows an example of an anomalous transaction processed by imageanalysis, consistent with the present disclosure. After a customercompletes a transaction that is different than expected based on avariety of factors based on information available about that customer,the image analysis system may cause an action to take place relating tothat anomalous transaction. For example, when a customer's purchasesitems are scanned or otherwise entered as part of a check-out process ata point of sale terminal 3301, point of sale terminal 3301 may generatesale information relating to each of the purchases. The point of saleinformation generated at point of sale terminal 3301 may includeidentifiers of the purchased items and other information such as price,quantity, brand, or similar objective measures.

After the point of sale data is generated at point of sale terminal3301, the point of sale data may be transferred to a system 3303designed to process the point of sale data produced at point of saleterminal 3301 and compare that point of sale data to demographic orhistorical information relating to the customer making the purchase todetermine whether an anomalous transaction has occurred. For example,this may include determining that the customer historically purchasedbroccoli, and the customer indeed purchased broccoli, confirming anexpected transaction. As another example, the customer typicallypurchases apples, and indeed the customer decided to purchase apples,confirming another expected transaction. Then, in what would bedetermined to be an anomalous transaction when compared to what thecustomer has historically purchased, the customer chose to purchasewhite bread. However, the customer's historical purchase behavior (e.g.,collected from past sales data, social network information, customerprofile information, etc.) suggests the customer normally purchaseswheat bread. After system 3303 has categorized this transaction ananomalous transaction, system 3303 may cause image capture equipment(e.g., capturing device 125 and image processing unit 130) to captureinformation related to the anomalous transaction.

For example, system 3303 may trigger capturing device 125 to captureimage data related to the anomalous transaction of the purchase of whitebread, and image data may be captured of the shelves where wheat breadand white bread are expected to be stocked in the retail store. System3303 may analyze the image data captured by capturing device 125 toidentify the current condition or status (e.g., in stock or not) ofwhite bread 2205 and wheat bread 3307. In this example, analyzing theimage data may determine a possible cause of the anomalous transaction:wheat bread 3307 is out of stock while white bread 3305 is currently instock. Based on the result of this analysis of the image data, system3303 may generate and cause transmission of, for example, a notificationto a handheld device 3309 used by a retail store associate or otherperson associated with the retail store. The notification may includeinformation regarding the anomalous transaction and a likely cause ofthe anomalous transaction (e.g., white bread 3305 is currently in stockon aisle 4 and wheat bread is currently out of stock in aisle 4).Additionally or alternatively to providing the notification, system 3303may additionally initiate actions that are manual or automated, e.g.,activating a robot or drone as demonstrated in FIG. 4A, FIG. 4B, andFIG. 4C.

FIG. 34 shows an example of a handheld device 3400 (e.g., a tablet,smartphone, etc.) displaying information generated to notify a user ofanomalous transactions, consistent with disclosed embodiments. Device3400 may have capabilities similar to those described in reference to,for example, FIG. 4A. Device 3400 may display various information to,for example, a store associate or other person associated with a retailstore. The information may include new anomalous transactions, such asthe unexpected transaction of wheat bread out of stock in aisle 4 asshown at 3401. The system may additionally show pending anomaloustransactions as shown at 3403. In this instance, the alert ornotification may show similar information as the information provided at3401 such as details regarding the anomalous transaction (e.g., product,product type, brand, price, etc.) and a location (e.g., a store aisle)associated with the purchased product, but may differ in display, e.g.,to demonstrate that the system or a store associate is in the process oftaking an action, thus making the resolution of the anomaloustransaction pending. A notification that has been resolved by a storeassociate or automated process as described above may show that theanomalous transaction has been resolved in notification 3405. Additionalinformation may be displayed to a store associate or other user in otherformats, including the, for example, as color-coded alerts (e.g., redrequiring attention, green signifying resolved, yellow representingpending, etc.). Additionally or alternatively, the alerts may includeinformation such as a date and/or time of an alert or other informationto aid a user to effectively manage the store's inventory.

FIG. 35 is a flowchart of an exemplary process 3500 for triggeringactions in response to point of sales data, consistent with the presentdisclosure. Process 3500 may be performed by at least one processor orat least one processing unit, such as processing unit 130 or processingdevice 202. It is to be understood that throughout the presentdisclosure, the term “processor” is used as a shorthand for “at leastone processor.” In other words, a processor may include one or morestructures that perform logic operations whether such structures arecollocated, connected, or dispersed. In some embodiments, anon-transitory computer readable medium may contain instructions thatwhen executed by a processor cause the processor to perform process3500. Further, process 3500 is not necessarily limited to the stepsshown in FIG. 35 , and any steps or processes of the various embodimentsdescribed throughout the present disclosure may also be included inprocess 3500.

In step 3501, process 3500 includes obtaining point of sale data from aretail store. The point of sale data may be generated at an automated orself-checkout system operated by a customer or an in-store associatechecking out a customer's purchases such as point of sale terminal 3301.The point of sale data may include information such as price, quantity,or similar objective measures of products purchased by a customer, asdiscussed above in reference to FIG. 33 .

In step 3503, process 3500 includes analyzing the point of sale data toidentify at least one anomalous transaction. Point of sale data mayconstitute at least one anomalous transaction when at least onepurchased product differs from expectations in respect to one or more ofbrand, periodicity, or amount of product being purchased.

In some embodiments, the at least one anomalous transaction may beidentified based on analysis of information relating to at least one ofhistoric shopping activities of a customer or demographic informationassociated with the customer. For example, the identification of ananomalous transaction may include analysis of the transaction based onone or more factors relating to the historical shopping activities(e.g., past purchases or preferences) of the customer involved in thetransaction or demographic information relating to the customer asdiscussed above in connection with FIG. 33 .

In some embodiments, the at least one anomalous transaction may includea detection of a customer purchasing a first product type rather than asecond product type, wherein the first product type and the secondproduct type are included in a common product category. In the examplediscussed above in connection with FIG. 33 , the first product type maybe white bread (the product ultimately purchased by the customer) andthe second product type may be wheat bread (the expected preference orhistorical purchase preference of the customer, and both the firstproduct and the second product type are bread types (i.e., a commonproduct category).

In some embodiments, a promotion may apply to the first product type,but not to the second product type. A promotion may include a specialdeal or discount applicable to the first product resulting in a reducedprice, rebate, free gift, or other incentive applicable to purchases ofproducts of the first product type. Accordingly, in such situations, thepromotion may cause first product type to be preferable to a customer ascompared to the second product type.

In some embodiments, a first promotion may apply to the first producttype and a second promotion may apply to the second product type,wherein the first promotion differs from the second promotion in atleast one aspect. The promotions may differ in price of the promotion,amount offered in each respective promotion, length of time thepromotion is being offered, or similar details differ so as to make onepromotion preferable or different over another.

In some embodiments, the customer may be expected to prefer products ofthe second product type over the first product type based on an analysisof historic shopping activities of the customer. The historic shoppingactivities may include previous purchases of one or both product types,the frequency of such purchases, the amount purchased in a giventransaction, or similar differences between historical purchases of thefirst product type and/or second product type.

In some embodiments, the customer may be expected to prefer products ofthe second product type over the first product type based on demographicinformation associated with the customer. Demographic informationassociated with the customer may include information such as age,gender, education, geographic location, occupation, or other categoricaldemographic information of the customer.

In some embodiments, the customer may be expected to prefer products ofthe second product type over the first product type based on purchasesof related products. The related products may include products producedby the same company, products in the same or similar categories ofproduct types (e.g., dinner rolls maybe considered to be a similarproduct to bread) or other relationships between product types.

In some embodiments, the identified at least one anomalous transactionmay correspond to a selection of a first product type for purchase,wherein the selection of the first product type occurs after an idleperiod, since a previous selection of the first product type forpurchase, of greater than a predetermined threshold. The predeterminedthreshold may be an expected periodicity based on the historicalshopping activity of the customer or the expected duration of use of thefirst product type. The predetermined threshold may be automaticallycalculated, predetermined based on historical data, or manually assignedto a customer based on information gathered about the customer.

In some embodiments, the identified at least one anomalous transactionmay include a selection of a first product type for purchase at a rateless frequent than predicted. The predicted rate of purchase may becalculated automatically by the system or may be determined and enteredmanually by a sales associate. The prediction may be based on historicalshopping activity or another metric that qualitatively or quantitativelydetermines a customer's rate or purchase periodicity.

In step 3505, in response to the identified at least one anomaloustransaction, process 3500 includes providing information configured tocause capturing of image data from the retail store. For example, afteridentifying at least one anomalous transaction, the disclosed systemsand method may determine, based on the transaction, possible locationsin the retail store of the product to be analyzed, the image capturingdevices that may be able to provide image data regarding the product ofthe anomalous transaction, and the amount of image data required toassess the cause of an anomalous transaction. For example, one or morecameras may capture image data of the product from one or more anglesand the captured image data may be analyzed to assess potential causesof the anomalous transaction.

In some embodiments, the image data may include representations ofproducts of both the first product type and the second product type. Forexample, the image data may include a still image or video ofrepresentations of the first product type and the second product type,and the system may analyze the image data to determine the amount ofproducts available in either or both of the first product type andsecond product type or the presence or lack thereof for one or bothproduct types.

In some embodiments, the image data may include a representation of textconveying a promotion associated with products of the first of productsor the second type. For example, the image data may include arepresentation of a display a sign or similar indication that the firstor second product includes a promotional offering of the product similarto the above determined promotion.

In some embodiments, the image data may include a representation of atleast one shelf associated with the retail store. For example, the imagedata may include visual or textual representations associating the shelflocated in the image data corresponds with the retail store.

In step 3507, process 3500 includes analyzing the image data relating tothe at least one anomalous transaction to determine at least onecondition associated with the at least one anomalous transaction in theretail store. For example, system 3303 may apply one or more imageanalysis techniques to identify shapes, text, and/or objects representedin the image data in order to determine at least one conditionassociated with the at least one anomalous transaction.

In some embodiments, analyzing the image data may include using amachine learning model trained using training examples to analyze theimage data and determine the at least one condition associated with theat least one anomalous transaction in the retail store. The trainingexamples may be included in an initial training set that may then beadopted, adjusted, and corrected, for example, to address any issues ofoverfit, underfit, or similar statistical modeling issues, and/or may beadjusted to a particular retail store, to a particular group of retailstores, to a particular region of retail stores, and so forth. Oneexample of a training example may include sample image data related to asample anomalous transaction, together with a label indicating acondition associated with the sample anomalous transaction. In oneexample, the trained machine learning model may be a visualclassification model configured to classify images and/or videos todifferent classes, each class may correspond to a different condition,and the image data may be classified to a particular class, therebydetermining the at least one condition.

In some embodiments, analyzing the image data may include calculating aconvolution of at least part of the image data thereby obtaining aresult value of the calculated convolution, in response to the resultvalue of the calculated convolution being a first value, including afirst condition in the determined at least one condition, and inresponse to the result value of the calculated convolution being asecond value, including a second condition in the determined at leastone condition, the second condition differs from the first condition. Aconvolution value based on the image data may include a portion or allpixels of the obtained image data. In analyzing the convolution modeland assigning a first value to the convolution results, the convolutionmodel may further determine a first condition as previously discussedfor a first product type and a second condition determined to be asecond convolution value based on a second condition, and may furtherdetermine that the first condition differs in a qualitative orquantitative amount from the second condition.

In some embodiments, the at least one condition may include anout-of-stock inventory status for products of the second product type.For example, an out-of-stock inventory status may indicate that aparticular product, such as wheat bread, which is of a bread producttype, is not in stock.

In some embodiments, the at least one condition may include a low-stockinventory status for products of the second product type. A low-stockinventory status may include an inventory of a product stocked lowerthan expected, lower than required to maintain a shelf fully stocked, ora preset amount of inventory as determined by a sales associate,manager, and/or automated system (e.g., via image analysis of shelvingunits).

In some embodiments, the at least one condition may include apromotional offer associated with the first product type. Thepromotional offer may include a reduced price, an extra product of thefirst type offered with purchase of the first product type, or similarlyattractive sale or promotion of a first product type that encourages acustomer to purchase products of the first type.

In some embodiments, the at least one condition may includenon-compliance with at least one planogram associated with the secondproduct type. The planogram may be a diagram or model that indicates theplacement of product types on shelves, and non-compliance may includedeviating from the planogram such that a particular shelving unit ordisplay specified by the planogram lacks one or more products of thesecond product type.

In some embodiments, the at least one condition may include a relativeshelf location between the second product type and the first producttype. The relative shelf location may specify a location (e.g.,adjacent, above, below, two shelves above, two shelves below, etc.) of ashelf of the second product type relative to a shelf of the firstproduct.

In step 3509, based on the analyzed image data relating to the at leastone anomalous transaction, process 3500 includes generating an indicatorassociated with the at least one condition. The generated indicator maybe displayed on a sales associate device such as a point of saleterminal, smart phone, tablet, or similar data processing device. Theindication may include information relating to the existence of ananomalous transaction, the product purchased in the anomaloustransaction, the product expected to be purchased in an expectedtransaction, the location of the product involved in the anomaloustransaction, the location of the product expected to be purchased in anexpected transaction, or image data relating to the anomalous product,an expected product involved in the anomalous transaction, and so forth.

In some embodiments, the indicator of the at least one condition may beconfigured to prompt initiation of an action to address the at least onecondition. The action may include a customer associate restocking theproduct type or ordering more of a particular product or product typefrom a supplier in order to correct the deficiency that causes the atleast one condition.

In some embodiment, the action may be performed by at least one employeeof the retail store. The employee may be engaged solely in this activityor may be one of many roles performed by the employee of the retailstore. In other embodiments, the at least one action may be taken byanother individual (e.g., a contractor) on behalf of the retail store.

In some embodiments, the action may be automatically performed by atleast one robotic unit associated with the retail store. The robot mayinclude drones and/or ground-based robots, and may be designed andprogrammed to carry out the functions necessary to correct conditionsand to return to a docking station or similar storage location or to becontinuously operating throughout the store fulfilling and correctingconditions developed due to anomalous conditions.

In some embodiments, the action may be taken by a user of the retailstore in response to an incentive offer made to the user. For example,the user may receive an incentive offer via a device (e.g., asmartphone, tablet, smartwatch, etc.) while shopping in the retailstore. The inventive offer may include a discount, promotion, rebate,free gift, etc., in exchange for the user taking the action to addressthe at least one condition.

Retail environments face challenges when demand and inventory shift asproducts become more or less popular. These changes may be due tovarious causes ranging from sales or promotions, price increases,advertising, and supply shortages, all of which may cause the supply anddemand of a given product to fluctuate. As supply and demand fluctuates,retail stores have sought solutions to manage inventory more accuratelyand efficiently.

To keep pace with evolving changes, a retailer store will need torespond to point of sale information that contradicts a status ofinventory available on shelves at a given point in time. For example,analysis of image data captured from one or more cameras in the retailstore may suggest that a given product or product type is currently outof stock. This out of stock condition may cause an inventory managementsystem to initiate an automated process to resolve the out of stockinventory by, restocking the shelf represented in the image data usingavailable reserve stock or placing an order for additional products.However, in some cases, the product or product type that has beendetermined to be out of stock is not, in fact, out of stock. The productmay be in stock in spite of captured image data suggesting the productis out of stock for a variety of reasons. For example, the product maybe shelved in atypical location, customer or sales associate feedbackmay have incorrectly indicated that the product is not in stock (e.g.,the customer of sales associate looked for the product in the wrongaisle, overlooked the product, etc.), a store associate may have alreadyrestocked a shelf after the image data was captured, and so forth. Ifpoint of sale information indicates that an out of stock condition doesnot exist before a restocking action is completed, the disclosed systemsand methods may override the initiated restocking action, thus enablingstore resources to focus on other tasks.

In an embodiment, a non-transitory computer-readable medium may includeinstructions that when executed by at least one processor cause the atleast one processor to perform a method for automatically maintaining aretail space. The non-transitory computer-readable medium may include amemory device of a server, laptop, personal desktop computer, smartphone, tablet, or similar data processing machine. Further detailsregarding the method are described below in connection with FIGS. 36,37, and 38 .

FIG. 36 depicts an image 3601 captured from a retail store, consistentwith the present disclosure. Image 3601 may be captured by a camerasystem such as those described in FIGS. 5A-5C and located at variouspositions in a retail store as described in FIGS. 6A-6C. Image data maybe captured in the retail store based on various triggers or schedules.For example, the disclosed systems and methods may capture image date atpredetermined time intervals, on demand due to customer feedback that aproduct is out of stock, after a sales transaction occurs indicatingthat the inventory of a product is low or depleted, or in response toother similar information causing the disclosed systems and methods toinitiate the capture of image data.

The captured image data may include or be associated with data (e.g.,metadata) that provides context for when and where an image (or videostream) was taken. This information may include one or more of a date,time, identifier of an image capture device, or a location (e.g., astore aisle number) of an image capture device that obtained the imagedata. Such information regarding the context of captured image data maybe stored in a database (e.g., database 140) in association with thecaptured image data. At least one processor (e.g., image processing unit130) may analyze the captured image data to determine that a producttype is out of stock by determining no products exist on a designatedshelf for the products. For example, as shown in FIG. 36 , wheat breadis not stocked on the shelving or display space designated for thisproduct type. Additionally, a time stamp of 11:00 AM indicates a capturetime when image 3601 was captured. Although only a time is depicted inFIG. 36 , image 3601 may include or otherwise be associated with otherinformation, such as date, location, etc., as discussed above.

When the disclosed systems and methods identify, based on image 3601,that a product is out of stock at a designated shelf or displaylocation, certain actions may be automated or initiated. For example,after processing and analyzing captured image data, the disclosedsystems and methods may initiate a manual or automatic action to restockan out of stork product. These actions may include activating a robot ordrone or transmitting a notification of the out of stock event to adevice associated with a retail associate.

FIG. 37 shows an example of using point of sale data to cancel an actionrelated to an out of stock condition in a retail store, consistent withthe present disclosure. As discussed in connection FIG. 36 , analysis ofimage 3601 has indicated that wheat bread is out of stock. In responseto an out of stock condition of wheat bread, the disclosed systems andmethods may generate information configured to cause an action to remedythe out of stock condition. In this example, the action may include themanual or automatic restocking of wheat bread in the retail store. Anautomated response may include the deployment of a robot or drone toaccomplish the restocking, and, a manual response may includetransmitting a notification to store associate providing instructions torestock the product (e.g., restock wheat bread at aisle 4).

However, after generating information configured to cause the action(e.g., a restocking event), the disclosed systems and methods mayreceive point of sale data from point of sale terminal 3703 indicatingthat a purchase of wheat bread occurred after image 3601 was captured,but before initiation of the restocking action. For example, as shown inFIG. 37 , wheat bread was purchased at 11:52 AM (i.e., after image 3601was captured at 11:02 AM). A particular product (wheat bread in thisexample) may be found to be in stock through analysis of point of salesdata for a variety of reasons. For example, the system may have analyzeda captured image and incorrectly concluded that a product is out ofstock due to poor image quality (e.g., a low light level), anobstruction (e.g., wheat bread was indeed available and on an expectedshelf but obscured by a shopping cart, a customer, a freestandingproduct display, a shipping container, etc.), the product was restockedafter the image was captured (e.g., wheat bread was temporarily stockedat a wrong location but returned to its proper shelf by a storeassociate after the image was taken), and so forth. Accordingly, inresponse to the received point of sales data, the system may generate anorder to cancel the action. In the case of a restocking order usingautomated restocking technology, the order may cancel a robot and/ordrone from proceeding to the shelf and cause the robot and/or drone tobe rerouted. For example, as shown at 3705, the order may cause a droneand robot to turn around and from an aisle where wheat bread is stockedand be deployed to another area of the retail store. In the case of arestocking order transmitted to a store associate with a task specifyinga product or product type for restocking, the order may cancel the task(e.g., the store associate may receive a cancelation message on a devicesuch as a tablet, smartphone, smartwatch, etc.).

FIG. 38 is a flowchart of an exemplary process 3800 for automaticallymaintaining a retail space, consistent with the present disclosure.Process 3800 may be performed by at least one processor or at least oneprocessing unit, such as processing unit 130 or processing device 202.It is to be understood that throughout the present disclosure, the term“processor” is used as a shorthand for “at least one processor.” Inother words, a processor may include one or more structures that performlogic operations whether such structures are collocated, connected, ordispersed. In some embodiments, a non-transitory computer readablemedium may contain instructions that when executed by a processor causethe processor to perform process 3800. Further, process 3800 is notnecessarily limited to the steps shown in FIG. 38 , and any steps orprocesses of the various embodiments described throughout the presentdisclosure may also be included in process 3800.

In step 3801, process 3800 includes receiving image data captured form aretail store. For example, the image data may be captured by a capturingdevice 125 that is fixed or mounted to a particular location in a retailstore. The captured image data may include one or more images (e.g.,still images or video).

In some embodiments, the image data may be representative of one or morelocations associated with the retail store. For example, the image datamay include representations of one or more aisles in the retail store.Further, process 3800 may determine a location in the retail storecorresponding to captured image data based on a known location of aninstalled camera (e.g., capturing device 125). Alternatively oradditionally, analyzing the image data may corroborate one or morelocations by analyzing text or images on signage and/or displays.

In step 3803, process 3800 includes analyzing the received image data todetermine that, at a capturing time, a first product type is out ofstock. This may include application of an image processing algorithm toidentify, for example, edges or shapes, and/or the use of textrecognition to identify text on product packaging and/or productdisplays (e.g., advertisements or price tags). The received image datamay include one or more of metadata, time stamps, or other informationto contextualize the received image data. The capturing time (e.g., atime stamp) may correspond to a time at which the image data wascaptured by an image capture device (e.g., image capture device 125).For example, as shown in FIG. 36 , image 3601 was captured at 11:00 AM,the first product type corresponds to wheat bread, and the first producttype is out of stock because items are vacant from the shelf. In someembodiments, the capturing time may include a date or a time, and inother embodiments, the capturing time may include a date and time.

In some embodiments, the image data may be representative of a shelf inthe retail store associated with products of the first product type. Forexample, the image data may include a representation of shelvesdesignated for a particular product (e.g., wheat bread). In some cases,process 3800 may determine that the image data is representative of aparticular shelf (e.g., a shelf designated for wheat bread) in theretail store based on one or more of metadata (e.g., an identifier of acamera that captured the image data) and/or analysis of the image datato identify one or more of signage, display stands, price tags, orproduct packaging (e.g., via text recognition).

In some embodiments, analyzing the received image data may include usinga machine learning model trained using training examples to analyze theimage data and to determine that, at the capturing time, the firstproduct is out of stock. The machine learning model (e.g., a trainedneural network) may apply training examples (e.g., previously capturedimages) to determine that the first product is out of stock.

In some embodiments, analyzing the received image data may includecalculating a convolution of at least part of the image data to therebyobtain a result value of the calculated convolution, and using theresult value of the calculated convolution to determine that, at thecapturing time, the first product is out of stock. A convolution valuebased on the image data may include any or all pixels of the obtainedimage data. In analyzing the convolution model and assigning a firstvalue to the convolution results, the convolution model may furtherdetermine a first condition for a first product type and a secondcondition determined to be a second convolution value based on a secondcondition, and may be further modeled to determine the first conditiondiffers in some qualitative or quantitative amount from the secondcondition.

In step 3805, in response to the determination that the first producttype is out of stock at the capturing time, process 3800 includesgenerating first information configured to cause at least one action inresponse to the determination that the first product type is out ofstock. The first information configured to cause at least one action inresponse to the determination that the first product type is out ofstock may include, activating automated systems such as a drone or robotand/or notifying an associate or manager of the condition. The at leastone action may include restocking the first product type with inventorynot currently on shelves, moving the first product type from anincorrect shelf to a correct shelf, reordering an out of stock product,or other corrective action.

In some embodiments, the first information may be configured toautomatically cause generation of one or more tasks associated with atask management system. For example, the one or more tasks may be rankedaccording to how immediate the one or more tasks must be completed, bythe periodicity required to keep a first product type in-stock andshelved, the popularity of the first product type to consumers, or anysimilar metric describing how quickly a task must be completed.

In some embodiments, the one or more tasks may include a request toextract products of the first product type from a current inventory andto restock a retail shelf with the extracted products of the firstproduct type. For example, the one or more tasks may cause an automatedresponse from a robot or drone and may include locating the firstproduct type in a secondary or storage location, retrieving the item andconfirming it matches the first product type requested or commanded bythe task, and moving the confirmed first product type from the inventorylocation to the tasks desired location (e.g., an appropriate shelf ordisplay).

In some embodiments, the one or more tasks may include a request togenerate a re-supply order for products of the first product type. Forexample, after determining that products of the first product type areout of stock, the disclosed system may generate a re-supply order toreplenish the first product type in order to obtain new inventory.

In some embodiments, the at least one action may include extractingproducts of the first product type from a current inventory andrestocking of a retail shelf with the extracted products of the firstproduct type. For example, the one or more action may cause an automatedresponse from a robot or drone and may include locating the firstproduct type in a secondary or storage location, retrieving the item andconfirming it matches the first product type requested by the action,and moving the confirmed first product type from the inventory locationto the tasks desired location (e.g., an appropriate shelf or display).

In some embodiments, the at least one action may include placing anorder for additional products of the first product type. For example,the system may transmit the order over one or more networks to a vendorfor fulfillment of new products.

In step 3807, process 3800 includes receiving point of sales data fromthe retail store, wherein the point of sales data indicates a purchaseof the first product type occurring after the capturing time, but beforeinitiation of the at least one action. For example, as discussed above,process 3800 may receive point of sale data from point of sale terminal3703 indicating that a purchase of the first product type (e.g., wheatbread) occurred after the capturing time (e.g., after image 3601 wascaptured), but before initiation of the at least one action (e.g., arestocking action). For example, as shown in FIG. 37 , wheat bread waspurchased at 11:52 AM (i.e., after image 3601 was captured), yet wheatbread was found to be in stock through point of sales data.

In some embodiments, process 3800 may include triggering the capture ofadditional image data in response to the received point of sale data.For example, the system may have analyzed a captured image andincorrectly concluded that a product is out of stock due to poor imagequality (e.g., a low light level, interruption of an image feed, etc.),an obstruction (e.g., wheat bread was indeed available and on anexpected shelf be obscured by a shopping cart, a customer, afreestanding product display, a shipping container, etc.), the productwas restocked after the image was captured (e.g., wheat bread wastemporarily stocked at a wrong location but returned to the proper shelfby a store associate after the image was taken), and so forth.Accordingly, to validate or verify the previously captured image data,process 3800 may capture additional image data from another location(e.g., a capturing device from another position) and/or at a later time.In some embodiments, the generation of the second information may bebased on an analysis of the additional image data. For example, inresponse to a first content of the additional image data, a firstversion of the second information may be generated, and in response to asecond content of the additional image data, a second version of thesecond information may be generated, the second version may differ fromthe first version. In another example, in response to a first content ofthe additional image data, the second information may be generated, andin response to a second content of the additional image data, generatingthe second information may be avoided. For example, when the additionalimage data shows that the first product type is not out of stock, thefirst second may be generated, and when the additional image dataconfirms that the first product type is out of stock, generating thesecond information may be avoided. In another example, when theadditional image data shows that the visual appearance of the firstproduct type changed, the second information may be generated to causean update to a visual recognition model associated with the firstproduct type in addition to canceling the at least one action, and whenthe additional image data shows that the visual appearance of the firstproduct type has not changed, the second information may be generated tocause the canceling the at least one action but not to cause the updateto a visual recognition model associated with the first product type.

In step 3809, in response to the received point of sales data, process3800 includes generating second information configured to cancel the atleast one action. In a restocking action using automated technology,when the point of sales data reveals that the first product type is infact in stock, the second information may cause a robot and/or drone toretain from proceeding to the shelf or may cause the robot and/or dronethat may be in route to the shelf to be rerouted. For example, as shownat 3705, the order may cause a drone and robot to depart from an aislewhere wheat bread is stocked and be deployed to another area of theretail store. In a restocking action where a store associate was taskedto restock wheat bread, the order may cancel the task. In some examples,step 3809 may generate the second information when the point of salesdata indicates that a number of purchases of the first product typeoccurring after the capturing time exceeds a predetermined threshold.For example, when the number of purchases of the first product typeoccurring after the capturing time does not exceed a predeterminedthreshold, the generation of the second information may be withheld. Insome examples, step 3809 may generate the second information when thepoint of sales data indicates that the purchase of the first producttype occurring after the capturing time is associated with a customerthat entered the retail store after the capturing time. For example, acustomer that entered the retail store before the capturing time mayhave picked the first product type before the capturing time, andtherefore purchases made by this customer may be discarded as a triggerto the generation of the second information. In some examples, step 3809may generate the second information when the point of sales dataindicates that the purchase of the first product type occurring afterthe capturing time occurred at least a predetermined time period afterthe capturing time. For example, the predetermined time period may beselected based on locations in the retail store associated with thefirst product type and the point of sale, based on a distance from thelocation in the retail store associated with the first product type andthe location in the retail store associated with the point of sale, andso forth.

In some embodiments, the second information may be configured to causeremoval from an instruction queue of the first information. The secondinformation may include, among other things, a command to a centralsystem to forego, delete, or cancel the command generated by the firstinformation.

In some embodiments, the second information may be configured toautomatically cause at least one of deletion or deactivation of at leastone previously generated task associated with the task managementsystem. Deletion or deactivation may involve causing the automatedaction to be terminated.

In some embodiments, the second information may be further configured tocause a second action different from the at least one action. Forexample, the second action may include an update to a planogramassociated with the retail store and the first product type. In anotherexample, the second action may include initiating capturing of one ormore images of one or more products of the first product type forupdating a visual recognition model associated with the first producttype. For example, the visual recognition model may be a machinelearning model trained using a dataset, and the visual recognition modelmay be updated by updating the dataset based on the captured one or moreimages of the one or more products of the first product type andretraining the visual recognition model. In another example, the visualrecognition model may be an online machine learning model trained usinga dataset, and the visual recognition model may be updated by using theone or more images of the one or more products of the first product typeas new training examples.

The foregoing description has been presented for purposes ofillustration. It is not exhaustive and is not limited to the preciseforms or embodiments disclosed. Modifications and adaptations will beapparent to those skilled in the art from consideration of thespecification and practice of the disclosed embodiments. Additionally,although aspects of the disclosed embodiments are described as beingstored in memory, one skilled in the art will appreciate that theseaspects can also be stored on other types of computer readable media,such as secondary storage devices, for example, hard disks or CD ROM, orother forms of RAM or ROM, USB media, DVD, Blu-ray, 4K Ultra HD Blu-ray,or other optical drive media.

Computer programs based on the written description and disclosed methodsare within the skill of an experienced developer. The various programsor program modules can be created using any of the techniques known toone skilled in the art or can be designed in connection with existingsoftware. For example, program sections or program modules can bedesigned in or by means of .Net Framework, .Net Compact Framework (andrelated languages, such as Visual Basic, C, etc.), Java, C++,Objective-C, HTML, HTML/AJAX combinations, XML, or HTML with includedJava applets.

Moreover, while illustrative embodiments have been described herein, thescope of any and all embodiments having equivalent elements,modifications, omissions, combinations (e.g., of aspects across variousembodiments), adaptations and/or alterations as would be appreciated bythose skilled in the art based on the present disclosure. Thelimitations in the claims are to be interpreted broadly based on thelanguage employed in the claims and not limited to examples described inthe present specification or during the prosecution of the application.The examples are to be construed as non-exclusive. Furthermore, thesteps of the disclosed methods may be modified in any manner, includingby reordering steps and/or inserting or deleting steps. It is intended,therefore, that the specification and examples be considered asillustrative only, with a true scope and spirit being indicated by thefollowing claims and their full scope of equivalents.

1. A non-transitory computer-readable medium including instructions thatwhen executed by a processor cause the processor to perform a method,the method comprising: obtaining an electronic shopping list associatedwith a customer, the electronic shopping list identifying a targetproduct; automatically determining that the target product isunavailable at a particular retail store; automatically identifying analternative product, available at the particular retail store, as apotential alternative to the target product; receiving at least oneimage representative of at least a portion of the particular retailstore wherein the at least one image includes a representation of thealternative product; and analyzing the at least one image to cause anextended reality appliance to present to a shopper a visual indicatorassociated with the alternative product in the particular retail store.2. The non-transitory computer-readable medium of claim 1, wherein theextended reality appliance is further caused to present a visualindication that the target product is not available at the particularretail store.
 3. The non-transitory computer-readable medium of claim 1,wherein the visual indicator includes a directional indicator forassisting in locating the alternative product within the particularretail store.
 4. The non-transitory computer-readable medium of claim 1,wherein the visual indicator includes an indication of a location in theparticular retail store where the alternative product is located.
 5. Thenon-transitory computer-readable medium of claim 1, wherein the visualindicator is presented as a first indicator type if substitution of thetarget product with the alternative product requires an authorizationfrom the customer, and wherein the visual indicator is presented as asecond indicator type, different from the first indicator type, ifsubstitution of the target product with the alternative product does notrequire an authorization from the customer.
 6. The non-transitorycomputer-readable medium of claim 1, wherein the extended realityappliance is a mobile communication device associated with the shopper.7. The non-transitory computer-readable medium of claim 6, wherein theat least one image includes a series of images, and the method furtherincludes analyzing the series of images to determine whether thealternative product is selected by the shopper.
 8. The non-transitorycomputer-readable medium of claim 7, wherein the method furthercomprises automatically updating the shopping list to reflect aselection of the alternative product type by the shopper.
 9. Thenon-transitory computer-readable medium of claim 7, wherein the methodfurther comprises sending a notification to the customer indicating aselection of the alternative product type by the shopper.
 10. Thenon-transitory computer-readable medium of claim 6, wherein the at leastone image is captured by an image sensor included on the mobile deviceassociated with the shopper.
 11. The non-transitory computer-readablemedium of claim 1, wherein the shopper is the customer.
 12. Thenon-transitory computer-readable medium of claim 1, wherein the shopperis a picker fulfilling an order associated with the customer and theelectronic shopping list.
 13. The non-transitory computer-readablemedium of claim 1, wherein the at least one image is captured by acamera mounted in the particular retail store.
 14. The non-transitorycomputer-readable medium of claim 1, wherein the extended realityappliance is a head-mounted device.
 15. The non-transitorycomputer-readable medium of claim 1, wherein automatically determiningthat the target product is unavailable at the particular retail store isbased on analysis of the at least one image.
 16. The non-transitorycomputer-readable medium of claim 1, wherein automatically determiningthat the target product is unavailable at the particular retail store isbased on analysis of inventory data associated with the particularretail store.
 17. The non-transitory computer-readable medium of claim1, wherein automatically identifying the alternative product is based onanalysis of the at least one image.
 18. The non-transitorycomputer-readable medium of claim 1, wherein automatically identifyingthe alternative product is based on analysis of inventory dataassociated with the particular retail store.
 19. A system for providingaugmented reality support to retail store pickers, the systemcomprising: at least one processing unit configured to: obtain anelectronic shopping list associated with a customer, the electronicshopping list identifying a target product; automatically determine thatthe target product is unavailable at a particular retail store;automatically identify an alternative product, available at theparticular retail store, as a potential alternative to the targetproduct; receive at least one image representative of at least a portionof the particular retail store wherein the at least one image includes arepresentation of the alternative product; and analyze the at least oneimage to cause an extended reality appliance to present to a shopper avisual indicator associated with the alternative product in theparticular retail store.
 20. A method for providing augmented realitysupport to retail store pickers, the method comprising: obtaining anelectronic shopping list associated with a customer, the electronicshopping list identifying a target product; automatically determiningthat the target product is unavailable at a particular retail store;automatically identifying an alternative product, available at theparticular retail store, as a potential alternative to the targetproduct; receiving at least one image representative of at least aportion of the particular retail store wherein the at least one imageincludes a representation of the alternative product; and analyzing theat least one image to cause an extended reality appliance to present toa shopper a visual indicator associated with the alternative product inthe particular retail store. 21-180. (canceled)